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INTRODUCTION 


Materiality and Invisibility 


Sean Cubitt, Daniel Palmer and Nathaniel Tkacz 


There is a story that the very first filter invented for Photoshop was the lens-flare. 
Usually regarded as a defect by photographers, lens flare is caused by internal 
reflection or scattering in the complex construction of compound lenses. It has the 
unfortunate effect of adding a displaced image of the sun or other light source, one 
that in cinematography especially can travel across the frame, and mask the ‘real’ 
subject. It also draws attention to the apparatus of picture-taking, and when used 
for effect transforms it into picture-making. The lens flare filter is said to have been 
added by Thomas Knoll, who had begun working on his image manipulation program 
as a PhD candidate at Stanford University in 1987, at the request of his brother 
John, a technician (and later senior visual effects supervisor) at Industrial Light 
and Magic, the George Lucas owned specialist effects house. Its first use would be 
precisely to emulate the photographic apparatus in shots that had been generated 
entirely in CGI (computer-generated imaging), where it was intended to give the illu- 
sion that a camera had been present, so increasing the apparent realism of the shot. 
The defect became simulated evidence of a fictional camera: a positive value. But 
soon enough designers recognised a second quality of the lens flare filter. By creat- 
ing artificial highlights on isolated elements in an image, lens flare gave the illusion 
of volume to 2D objects, a trick so widely disseminated in the 1990s that it became 


almost a hallmark of digital images. In this second use, the once temporal character 
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of flares—as evidence that a camera had ‘really’ been present—became instead a 
tool for producing spatial effects. That is, they are used not for veracity but for fan- 
tasy, evidence not of a past presence of cameras, but of a futurity toward which they 
can propel their audiences.! 

The history of lens flares gives us a clue about the transitions between analogue 
and digital in visual media that lie at the heart of this collection. The movement is 
by no means one-way. For some years, cinematographic use of flare in films like 
Lawrence of Arabia (David Lean, 1962) had evoked extreme states of consciousness, 
even of divine light, and with it the long history of light in human affairs. We can only 
speculate about the meanings of light during the millennia preceding the scriptures 
of Babylon and Judaea. Somewhere around a half a million years ago human ances- 
tors tamed fire (Diamond 1997: 38). Separating light from dark instigates creation 
in Genesis. Before the fiat lux, the Earth was ‘without form and void’. Formlessness, 
Augustine’s imaginary interlocutor suggests (Confessions XII: 21-2; 1961: 297-300), 
already existed, and it was from this formlessness that God created the world, as 
later he would create Adam out of a handful of dust. For Erigena in the eighth cen- 
tury, omnia quae sunt, lumina sunt: all things that are, are lights. In the De luce of 
the twelfth century divine Robert Grosseteste: 


.. light, which is the first form in first created matter, extended 
itself at the beginning of time, multiplying itself an infinity of times 
through itself on every side and stretching out equally in every direc- 
tion, dispersing with itself matter, which it was not able to desert, to 
such a great mass as the fabric of the cosmos. (Grosseteste, quoted in 
MacKenzie 1996: 26) 


As the analogue to Divine Light (which, Anselm had lamented, was by definition 
invisible), light pours form from God into creation. While mystics sought to plunge 
into the darkness of unknowing in order to find their way back to the creator, 
Grosseteste attributed to light the making of form (space) as well as the governance 
of the heavens (time) to provide a model scientific and theological understanding 
of light’s role in the moment of creation. The word ‘light’ can scarcely be uttered 
without mystical connotations. Understanding light’s connotations of yearning for 
something more, something beyond is important because these ancient and theologi- 
cal traditions persist; and because they also provide a persistent counterpoint to the 
rationalist and instrumental modernisation of light, at once universal and deeply 


historical, whose transitions from one technical form to another are our subject. 
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Digital light is, as Stephen Jones points out in his contribution, an oxymoron: 
light is photons, particulate and discrete, and therefore always digital. But photons 
are also waveforms, subject to manipulation in myriad ways. From Fourier trans- 
forms to chip design, colour management to the translation of vector graphics into 
arithmetic displays, light is constantly disciplined to human purposes. The inven- 
tion of mechanical media is a critical conjuncture in that history. Photography, 
chronophotography and cinematography form only one part of this disciplining. 
Photography began life as a print medium, though today it is probably viewed at 
least as much on screens. In common with the lithographic printing technique that 
preceded it by a scant few decades, photography was based on a random scatter 
of molecules, a scatter which would continue into the sprayed phosphors lining the 
interior of the first television cathode ray tubes. Mass circulation of photographs 
required a stronger discipline: the half-tone system, which imposed for the first time 
a grid structuring the grain of the image. Rapid transmission of images, required 
by a burgeoning press industry, spurred the development of the drum scanner that 
in turn supplied the cathode ray tube with its operating principle. But this was not 
enough to control light. From the Trinitron mask to the sub-pixel construction of 
LCD and plasma screens, the grid became the essential attribute of a standardised 
system of imaging that constrains the design and fabrication of chips, the software 
of image-manipulation software, and the fundamental systems for image transmis- 
sion. This is the genealogy of the raster screen that dominates visual culture from 
handheld devices to stadium and city plaza screens. As Sean Cubitt, Carolyn L 
Kane and Cathryn Vasseleu investigate in their chapters, the control of light forms 
the foundation of contemporary vision. 

In this collection, we bring together high profile figures in diverse but increas- 
ingly convergent fields, from academy award-winner and co-founder of Pixar, Alvy 
Ray Smith to feminist philosopher Cathryn Vasseleu. Several of the chapters 
originated in a symposium in Melbourne in 2011 called ‘Digital Light: Technique, 
Technology, Creation’.? At that event, practitioners and theorists discussed the rela- 
tionships between technologies (such as screens, projectors, cameras, networks, 
camera mounts, objects set up on rostrum cameras, hardware and software) and 
techniques (the handling, organisation, networking and interfacing of various kinds 
of electronics, other physical media, people, weather and natural light, among oth- 
ers). This interest in the creative process has flowed into this book, based on the 
hunch that artists (and curators and software engineers) proceed by working on and 
with, but also against the capabilities of the media they inherit or, in certain cases, 


invent. If our first concern is with the historical shaping of light in contemporary 
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culture, our second is how artists, curators and engineers confront and challenge 
the constraints of increasingly normalised digital visual media. In this regard, cur- 
rent arguments over the shape of codecs (compression-decompression algorithms 
governing the transmission and display of electronic images) under the HTML5 revi- 
sion to the language of the World Wide Web need to extend beyond the legal-tech- 
nical question of proprietary versus open source standards (Holwerda 2010). These 
codecs are the culmination of a process (now some fifty years old) of pioneering, 
innovating, standardising and normative agreement around the more fundamen- 
tal question of the organisation and management of the image—and by extension, 
perception. 

A unique quality of this edited collection is the blending of renowned artists and 
practitioners with leading scholars to address a single topic: the gains and losses 
in the transition from analogue to digital media. Even as we argue that the crude 
binary opposition between digital and analogue stands in need of redefinition, 
we also propose that fundamental changes in media are symptoms and causes of 
changes in how we inhabit and experience the world. The book opens with essays 
on the history and contemporary practice of photography and video, broadening 
out to essays on the specificity of digital media. The book constantly moves from 
an artist or practitioner to a historian or scholar, and then to a curator - in this 
regard we are delighted to have the participation of leading curators of media art, 
Christiane Paul and Jon Ippolito. While various art pieces and other content are 
considered throughout the collection, the focus is specifically on what such pieces 
suggest about the intersection between technique and technology.* That is, the col- 
lection emphasises the centrality of use and experimentation in the shaping of tech- 
nological platforms. Indeed, a recurring theme is how techniques of previous media 
become technologies, inscribed in both digital software and hardware (Manovich 
2001; 2013). Contributions include considerations of image-oriented software and 
file formats; screen technologies; projection and urban screen surfaces; histories of 
computer graphics, 2D and 3D image editing software, photography and cinematic 
art; and transformations of light-based art resulting from the distributed architec- 
tures of the internet and the logic of the database. 

If we were to single out a moment of maximum technical innovation, it might well 
be the mid-nineteenth century. Geoffrey Batchen (2006) considers William Henry 
Fox Talbot’s contact prints of lacework, noting that they were featured at soirées at 
the house of Charles Babbage, inventor of the Difference Engine and forefather of 
modern computing. In the same room, Babbage had on display an intricate silk por- 


trait of Joseph Marie Jacquard, whose silk loom was driven by the punch cards that 


Introduction 11 


the aptly-named Ada Lovelace would use as the first storage device for Babbage’s 
computer. One of Batchen’s points is that in many respects photography has always 
been digital. What we can also learn from his analysis is that innovation seems often 
to derive from social ‘scenes’, a thesis which resonates with the chapters by Alvy 
Ray Smith and Stephen Jones—who, despite their differences, share an under- 
standing of the vital importance of social as well as technological networks in the 
making of art. 

In his remarkable chapter, Smith outlines the development of the pixel, down- 
playing the importance of output devices such as screens, citing their extreme vari- 
ability and frequent unreliability, their coarse conversion of vector graphics to bit- 
maps and their inability to display the full gamut of colours existing in virtual state 
in the computer. Rather than diminish the importance of screen aesthetics, such 
statements clarify what is at stake in the practical specificity of different screens. 
This alone reveals the inadequacy of generalized accounts of digital aesthetics. In 
fact there is no single, universal and coherent digital aesthetics but a plurality of dif- 
ferent approaches, deployments and applications. For if, on the one hand, there is a 
tendency towards software standardisation, on the other there is radical divergence 
in technique, and radical innovation in technologies and their assemblage into new 
apparatuses. These developments must drive us to pay far more detailed attention 
to the materiality of artworks now than in the recent past, when what a film or tele- 
vision programme was made of scarcely signified, since most works were made of 
the same things and in the same way as all the others. This characteristic dialectic 
of standardisation and innovation is integral to building the library of techniques 
and technologies on which artists and others may draw. One of the most intrigu- 
ing of these is colour management, which Smith relates in an anecdote about his 
invention of the HSV (hue-saturation-value) colour space. Smith found that while 
standard RGB (red-green-blue) system allowed the mixing of the optical primaries, 
an efficient way of coding colour both for the red, green and blue receptors in the 
human eye, it is basically a two-dimensional space, based on physical optics, not the 
psychological optics of human perception. HSV, a three-dimensional colour space, 
allowed users to make a darker orange or a paler red by changing the value (roughly 
the brightness) of the defining hue and saturation bringing it closer to the intuitive 
way we mix colours like brown or pink in the physical world, and the experience of 
painters and designers. 

This insight into creative practice in software engineering opens up a whole new 
set of relations around the figure of the artist-engineer. The history of video and 


digital art is full of such characters: if anything, the process is accelerating, even as 
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the norms of dominant software seem to become more and more entrenched. Smith’s 
anecdote also suggested that the critical principle—that nothing is forced to be the 
way it is—holds good also of engineering, and that familiarity with and faith ina 
particular solution can become an obstacle to both the fluid use of the tool and the 
development of new tools. In Smith’s case, the older colour model restricted users’ 
ability to generate the effects they wanted, guiding them to the limited palette of 
colour privileged by the model. Creative software, whether produced in the studio or 
the lab, provides users with tools only the most sophisticated would have realised, in 
advance, that they needed. In this instance we also learn that the specific networks 
of devices and people established to make a particular work, in software or moving 
image art, are not necessarily stable or harmonious. Instability and ephemerality 
are near-synonyms of twenty-first century media and media arts, driven as they are 
by the dialectic between standardisation and innovation. 

As Tofts argues of Joel Zika’s digital photographs, attuning ourselves to digital 
perception creates a discomfort, out of which other perceptions and other prac- 
tices can arise. Similarly, Kane’s archaeology of artists at Bell Labs in the formative 
years of computer arts in the early 1960s demonstrates both the value of artistic 
creation in blue-skies research and the value of research free from governmental 
and commercial pressure. It also gives due prominence to Lillian Schwartz, one of 
many women who, since Ada Lovelace, have played a foundational role in the digital 
media. Intriguingly, it adds to these concerns the discovery of a perceptual rather 
than physical production of colour at the very beginnings of digital animation, in 
experimental artworks that produced optical colour from black and white when han- 
dled in subtle and swift succession. The old dialectic between Newtonian optics and 
Goethe’s physiological and psychological approach to colour, though resolved earlier 
for print and dye media some years earlier, remained in play in the 1960s in experi- 
ments which would then become normative in the good-enough colour management 
systems developed for MPEG and related video transmission standards. 

Another dialectic emerges in recent writings of Victor Burgin, whose contribu- 
tion to the conference from which this book derives has been published elsewhere 
(Burgin 2014). For Burgin, who has always situated his art practice in relation to 
the media environment, virtual cameras are a logical extension of the photographic 
and later video works with which he made his name. Burgin retains a serious and 
methodical eye not only for the technical detail (for instance, where the panoramic 
camera’s footprint appears in a digital cyclorama) but also a sense of the paradoxes 
inherent in the concept of the single, whole, and gestalt image which can be taken 
in, as Michael Fried (1992) once argued, in a single glance. One paradox of Fried’s 
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unified image is immediately discernible in panoramas, which surely fall under the 
concept of ‘image’, but where the image is not apparent or intelligible without specta- 
torial movement. In digital panoramas, a mobile viewpoint is always implicit. Today, 
artists are provided with such a mobile viewpoint in the ‘virtual camera’ embedded 
in the workspace of their image processing software. The end user or viewer, espe- 
cially in the age of computer video games, is surely entitled to expect one too. The 
dialectic between standardisation and innovation also re-emerges in Burgin’s work 
bir okuma yeri/a place to read (2010), a virtual fly-through of a once-iconic Istanbul 
coffee house, now moved to another site and in disrepair. Burgin’s piece recon- 
structs the building as a 3D graphic, using sprites (photographic surfaces applied to 
3D objects) derived from photographs of the surviving parts of the building. Burgin 
has returned the house to its gardens overlooking the Bosphorus, but the result is 
an uncanny dialectic between the mobile virtual camera and the unmoving photo- 
graphed background leaves and waters. Burgin made a point of using off-the-shelf 
software for this project, suggesting that the dialectic of standardisation and inno- 
vation can become the principle of a work of art, not least one concerned with the 
historical process in which they act out their intertwined destinies. 

Such dialectical disjunctures motivate, even necessitate, creative acts taking 
agency back from automated systems and default values. One example in Christiane 
Paul’s chapter is SVEN, the Surveillance Video Entertainment Network (http:// 
deprogramming.us/ai), whose project, according to their website, asks ‘If computer 
vision technology can be used to detect when you look like a terrorist, criminal, 
or other “undesirable”’—why not when you look like a rock star?’ Using a variant 
of recognition software, this closed circuit installation tracks people’s movements, 
and matches them with a library of rock star moves and poses, interpolating the 
CCTV capture with music video footage, encouraging both a voyeuristic fascination 
turned playful, and a performative attitude to the ubiquitous surveillance of contem- 
porary society. The goals of such practices are not normative and standardisable but 
dissenting and in almost every instance productive of alternatives. Such works are 
political in the sense that they create new conditions of possibility. In this sense vir- 
tual art produces the virtual, with its root-word virtus, strength, or potential, from 
its root-word power, potentia, the ability to act, that is, to make the virtual actual. 
As the realm of potential, politics is the power to create possibilities, to unpick the 
actual in order to create events in which matters change. In changing their own 
materials, the media arts model the construction of possibility, the construction 


of the open future. Against such virtual capabilities, efforts to de-materialise the 
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supposedly post-medium media are effectively attempts to stay within the consen- 
sual, agentless, eventless horizon of normal culture. 

Paul’s, Jon Ippolito’s, Scott McQuire’s and Daniel Palmer’s chapters touch 
on the topic of another specific adaptation of a contemporary medium, that of sur- 
veillance, and its new form as the mass surveillance of big data through always- 
on social media portals. They raised the possibility that a distinguishing feature 
of digital networks is, in David Lyon’s (1994) phrase, the ‘electronic panopticon’. It 
is certainly the case that network media provide governments and even more so 
advertisers with extremely detailed accounts of human behaviour. As Ippolito points 
out, the metaphor of light as information and truth is integral to surveillance. This 
metaphor is, we might add, common to both the surveyors and the surveyed—com- 
mon to those who seek to use it for government or profit as well as those who want 
to preserve an imagined privacy, a personal space of truth, safe from the powers of 
surveillance. In this way the question of the specificity of digital as opposed to ana- 
logue light is exposed to a further critique: if analogue photography claims a privi- 
leged indexical relation to the real, does that anchor it in regimes of surveillance, as 
John Tagg (1993) argued decades ago? Does the distributed and dispersed nature of 
digital light free it from that objectivising and instrumental destiny? 

Batchen’s (2006) argument, that photography is already digital in its earliest 
beginnings, is echoed by Jones’s reference to the switch as the fundamental digital 
tool. In binary computing, switches ensure that electrical current either flows or 
does not, providing the physical basis for the logical symbols 0 and 1. Reflecting on 
the quantum nature of physical light, Jones emphasises the concept that light moves 
in discrete packets (‘quanta’) or particles. Yet there remains the doubt expressed by 
Palmer and McQuire that in their sheer numbers as well as the material aesthetic 
of devices, images are becoming data, and subject to the same statistical manipula- 
tions and instrumental exploitation as the statistical social sciences that emerged 
contemporaneously with photography in the nineteenth century. 

To reduce the complex interactions of digital and analogue into a simple binary 
opposition is to grasp at essences where none can be relied on. Both the speed of 
innovation, and the unstable relation between bitmap and vector graphics and dis- 
plays suggest that there is no essence of the digital to distinguish it from the ana- 
logue, and that instead we should be focussing as creators, curators and scholars 
on the real specificity of the individual work or process we are observing. However, 
important recent work in software studies (for example, Fuller 2005) disputes the 
implication that the speed of innovation implies that computing inhabits a world of 


perpetual progress, arguing that it is shaped by corporate interests rather than a 
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pure logic of computing, and that it drags along with it redundant engineering prin- 
ciples (a familiar example is the persistence of the 1872 Scholes QWERTY typewriter 
keyboard into the foreseeable future). Smith, however, is more optimistic, arguing 
the opposite case. In any event, the software studies pioneered during the 2000s 
are beginning to be matched by studies of hardware. In software studies, the once 
monolithic concept of code is being broken up into discrete fields: codecs, operating 
systems, algorithms, human-computer interfaces and many more. Hardware stud- 
ies likewise point us towards the functioning of both individual elements in digital 
media—chips, amplifiers, displays and so on—and the often unique and frequently 
evolving assemblies that constitute the working platform for specific projects. The 
contributions here, notably Terry Flaxton’s chapter, provide refreshing evidence 
that the inventiveness and creativity of artists is integral to technical innovation, 
and to assessing not just cost and efficiency but such other values as the environ- 
mental and social consequences of technological ‘progress’. It became clear that, 
faced with such dedicated craft, at the very least, a critic should pay precise and 
careful attention to the actual workings of moving image media in the twenty-first 
century, now that the old stabilities of twentieth century technology and institutions 
are gone. Only in such attentiveness will we avoid both film studies’ prematurely 
assured belief in the specificity of digital versus analogue media, and art theory’s 
equally assured dismissal of medium specificity. If this book contributes to an 
awareness of these challenges, while also widening awareness of the richness of 
contemporary digital media arts, it will have done well. 

Of course, many visual technologies have faded into oblivion (Huhtamo and 
Parikka 2011; Acland 2007) and even in our own era of digital invention, once trum- 
peted technologies like immersive virtual reality and the CD-ROM have passed on 
to the gnawing criticism of the mice. Already, in the period of the historical avant- 
gardes, it had become apparent that every advance was all too readily assimilated 
into the gaping maw of advertising and commercialism, even as the vanguard of art 
found itself increasingly severed from its audience by the very difficulty of its inno- 
vations (Burger 1984). The same appears to be true of digital media: every technique 
is open to exploitation by a ravenous machinery devoted to the churn of novelty. 
Meanwhile, the old stability of film and television passes into a new instability. In 
some respects, every film is a prototype, but between the early 1930s and the early 
1990s, production techniques, management and technologies remained more or less 
stable. Today, however, each film assembles a unique concatenation of tools, from 
cameras to software. We are entering a period of extreme specificity, where the 


choice of editing software or the development of new plug-ins changes the aesthetic 
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of each film that appears. These cycles of rapid invention, depletion and abandon- 
ment make any statement about digital aesthetics moot. 

Thus the differences between analogue and digital devices can be overstated. 
When photons trigger the oxidation of silver salts in traditional photography, a 
by-product is the release of an electron. When photons trigger the optoelectronic 
response in chip-based cameras, it is the electrons that are captured, but in many 
respects the chemistry of the two operations is similar. Both require periods of 
latency, the one awaiting chemical amplification in the developing process, the other 
the draining of electrons from the chip prior to the next exposure, a feature that 
makes clear that there is no difference to be sought in the constant visibility of ana- 
logue as opposed to digital images. Meanwhile darkroom technicians have manip- 
ulated images with all the subtlety and imagination of Photoshop since the 1870s 
(Przyblyski 1995). Light itself may well be eternal, and its handling historical, but 
we should not seek radical change where there is none. The movement of history, 
especially the history of our sensual appreciation of the world, transforms itself far 
more rarely and slowly than our politics. 

At the same time we should not understate the significance of even small adapta- 
tions, as the case of lens-flare should remind us. Just as every film is a prototype, 
so every print of a film or photo is unique, a point made poignantly in John Berger’s 
(1975) anecdote of the treasured torn photograph of his son carried by an illegal 
migrant worker. Digital images are no less specific, carrying the scars of their suc- 
cessive compressions and decompressions, the bit rot attendant on copying and the 
vicissitudes of storage, and the unique colour depth and resolution of the screens 
and printers we use to access them. Such qualities belong to the particularity of 
making art with light-based technologies, and with the condition of viewing them. 
In this they bear highly time-bound and materially grounded witness to the condi- 
tions of making, circulation and reception, and thus to the fundamental instability 
of light itself. 

There is no absolute rift between the material practice of managing light and 
its emblematic function as the symbol of divinity, reason or knowledge. There is, 
however, a dialectic between symbolic and material functions of light played out 
in every image, a dialectic that comes to the fore in many works of art made with 
photomechanical and optoelectronic tools. One of the great terrains of this struggle 
is realism, that mode of practice that seeks in gathered light the evidence of an 
extra-human reality. It is striking that the schools of speculative realism and object- 
oriented philosophy, with their insistent ontology of things, should arise in a moment 


when digital media have ostensibly driven a wedge between the human sensorium 
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and its surrounding world. Where once the existence of divine providence proved 
the worth, and indeed the existence, of human beings, since the nineteenth century 
inventions of technical visual media, it is external reality that proves to us that we 
exist: as the alienated observers, the subjects, of a reality that appears not only to us 
but for us. With digital media, and in parallel with the development of chemicals sen- 
sitive to other wavelengths, the world no longer necessarily appears in images in the 
same visual form that it would have to a real human observer at the same place and 
time. To a certain extent, all images today, analogue and digital, have the character- 
istics of data visualisations, gathering photons or other electromagnetic waveforms 
from X-ray to ultraviolet, and indeed energy forms that baffle comprehension (Elkins 
2008; Galison 1997). What is at stake in the debates over realism is a quarrel over 
the status not of reality but of the human. 

The light of God, of reason, of science, of truth: light’s metaphorical power is 
undimmed by the material practices in which it is embroiled. Whether invoking the 
brilliance of creation or an impossibly bright technological future, the practice of 
light in the hands of engineers, artists and producers generally is a constant strug- 
gle between boundless, uncontrolled effulgence and the laser-accurate construction 
of artefacts that illuminate and move their viewers. This collection undertakes a 
snapshot of this struggle at a moment of profound uncertainty. The chapters that 
follow enquire, through practice and thinking, practice as thinking and thinking as 
practice, into the stakes and the opportunities of this extraordinary moment. 
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Notes 


1. 


For J.J. Abrams, here discussing his Star Trek film, lens flare belongs to the future: 

They were all done live, they weren’t added later. There are something about those 
flares, especially in a movie that can potentially be very sterile and CG and overly 
controlled. There is something incredibly unpredictable and gorgeous about them. 
... Our DP would be off camera with this incredibly powerful flashlight aiming it at 
the lens. It became an art because different lenses required angles, and different 
proximity to the lens. Sometimes, when we were outside we’d use mirrors. ... We had 
two cameras, so sometimes we had two different spotlight operators. When there was 
atmosphere in the room, you had to be really careful because you could see the beams. 
... [It] feels like the future is that bright. (Woerner 2009) 

Abrams’ flares takes us back to the affective level of image capture, and to the artful 
engineering of analogue flares for the live elements in a CG environment, not for 
veracity but for fantasy, evidence not of a past presence of cameras, but of a futurity 
toward which they can propel their audiences. These versions of lens-flares indicate 
that to some extent all photography depicts not only objects but the apparatus of 
imaging. Archivists well know that a plate or frame carries the evidence of its making, 
and indeed of the ‘archival life’ (Fossati 2009) that it has led since the shutter closed. 
This materiality is integral to the aesthetics of photography, that diverse family of 
technologies which use light for mark-making (Maynard 1997: 3). 


Supported by the Australia Research Council Discovery Project, ‘Digital Light: 
Technique, Technology, Creation’ was held at The University of Melbourne over two 
days, 18-19 March 2011. Our speakers were, in alphabetical order, Geoffrey Batchen, 
Victor Burgin, Steve Dietz, Jon Ippolito, Stephen Jones, Alex Monteith, Christiane 
Paul, Jeffrey Shaw, Alvy Ray Smith, Van Sowerwine, and Lynette Wallworth. Another 
account of the event is given in Cubitt et al (2012). 


In December 2005, Adobe completed its amalgamation with Macromedia, establishing 
an effective monopoly of industrial 2D design applications. A little over a month 
later in January 2006, Autodesk, the market-leader in computer-aided design and 
manufacture, engineering and architectural visualisation software, acquired Maya, 
the leading 3D graphics package, adding it to the Discreet workflow and effects 
manager acquired in 1999 and its internally developed and highly successful 3ds 
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Max 3D software. In 2008 it acquired Softimage, the 3D company co-founded by 
pioneer immersive VR artist Char Davies. During the same period, Apple’s Final Cut 
suite of video tools has come to dominate the professional market for moving image 
post-production. Our case is that the carefully matched workspaces and workflow 
management of these dominant companies, along with their consequent ubiquity in 
training courses, produces a normalisation of visual digital culture of a depth and 
reach, which is unprecedented. We do not make any case here about the business 
models employed. Rather we are concerned that the experience of using Adobe, 
Autodesk and Apple frames the vast majority of what we think of as creative practice 
in the second decade of the twenty-first century; and that the practices which they 
encourage—as all software does, by making some things easier and others harder to 
achieve—are becoming the ordinary frame through which our visual culture works. 
Standards such as codecs are agreed, in the main, between hardware, software and 
telecommunications corporations (although nominally by governments, and in certain 
key instances by professional bodies of independent engineers). It would be rash to 
argue that codec standardisation constitutes a conspiracy to blinker our perceptions. 
On the other hand, it would be naive to deny that they inform the way we see and feel 
space, movement and time, the operations of light and colour (Mackenzie 2008; Cubitt 
2008). Codecs establish an aesthetic: they are the frames through which we observe 
and construct the world and our experience of it. Without necessarily meaning to, and 
left to its own devices, such normalisation would stifle the innovation and creativity 
it was designed to encourage. Our attempt at the symposium was to investigate how 
people both invent new media like these; and how they and others then chip away at 
the normative tendencies inherent in technical innovation processes to create new 
uses and applications, which in turn may lead to new technologies, and to new modes 
of inhabiting the world. 


4. Such practices do not simply break up the over-confident binary of analogue versus 
digital: they also problematize art historian Rosalind Krauss’ argument regarding the 
post-medium condition. Krauss’ target is the orthodox modernist aesthetics of Clement 
Greenberg who argued that each medium (painting, sculpture, music, etc.) has its own 
internal logic, and that the work of the artist was specific to the logic of the medium. 
Greenberg believed that the painter should do painterly things with paint, not tell 
stories, or dramatise events. Krauss proposes instead that Greenberg’s media are 
sets of conventions, ‘distinct from the material properties of a merely physical object- 
like support’ (Krauss 2000: 27), and that there is therefore no ontological connection 
between the physical medium—of paint, for example—and the kinds of activity that 
can or should be undertaken with it. The talismanic place of flatness in Greenberg’s 
account of painting is thus wholly conventional, and painters should feel free to tell 
stories or make three-dimensional objects, and use any medium in any way they see fit. 
By inference, for Krauss there is no ontological characteristic defining digital media; 
there are no distinguishing features separating digital and analogue, nor verbal and 
visual or any other media. Against Krauss, however, and against the concept of a post- 
medium condition, we argue that there are indeed ontological properties and unique 
aesthetics attached to specific devices of both digital and analogue media. These 
arise with particular choices such as what analogue film stock or what digital codec 
to employ, and the design of tools shared by both, such as wide-angle or macro lenses. 


CHAPTER 1 


A Taxonomy and Genealogy of 
Digital Light-Based Technologies 


Alvy Ray Smith 


At a recent SIGGRAPH, in receiving the computer graphics organization’s highest 
award, Rob Cook of Pixar said essentially this: ‘It took us almost forty years, but 
we've done it. Now what do we do?’ The accomplishment he referred to was this: we 
had, over the course of four decades or so, managed to extract competent film and 
video production from sheer computation. Add to this the more-or-less simultane- 
ous desk-top publishing revolution and later the video game explosion and you have 
it: the creation of pictures—including moving pictures and interactive ones—from 
mere bits, the replacement of all earlier picturing media by just one, that of com- 
putation. I suggest this accomplishment is what the editors of this book call ‘digital 
light’. We tended to call the process, especially late in the game, the ‘digital conver- 
gence’. As a participant and observer throughout those forty years, I hope to assist 
in the project of understanding and criticizing digital light-based technologies and 
techniques. I use my own experiences as a backbone and timeline from which I hang 
the genealogy of the concepts involved. 

An additional anecdote serves as a teaser for a topic I address later in the paper. 
At another awards ceremony, that of the technical Academy awards in 1998, Richard 
Dreyfus, the great actor, was presiding. (Aside: The digital Academy awards are 


just like the artistic Academy awards shown on the television every year—the same 
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glamour, the same actors, the same gowns and tuxedos and jewels and limos, also 
in Hollywood. They are just considered not very interesting to the American pub- 
lic, so are not broadcast. The actors contribute their precious time to this event in 
homage to the technologies they know underlie their field. It was Richard Dreyfus 
who had the honours that year.) He paid the usual lip service to the joining together 
of the artistically creative and the technically creative to produce the cinema, but 
then added: ‘We will march together into the next century.’ Then looking directly at 
our table, he said, ‘Note that I said together you people from Pixar, who made Toy 
Story.’ A nervous titter spread through the audience, especially the actors, the fear 
so expressed being the one that computers would take over the job of actors any day 
now. Although Rob Cook said we had done it, perhaps the holy grail is still out there: 
simulation of live action, in particular of human actors. I return to this problem 
later: Have we yet simulated human actors? Is live action about to be encompassed? 
Has it already? 

Another topic I emphasize is the dangers of ‘lock in’. In the early exciting days of 
new software, with the urge for primacy at our backs, we were prone to make quick 
decisions about fundamental issues that might later haunt us. The easy idea that 
looked so good at first blush got locked in by so much code written atop it, and then 
could not be removed. One of these I discuss is the rectilinear image. It is so wired 
into the software systems of the world that it is hard to think otherwise. The notion 
of non-rectilinear images, or sprites, is there to be exploited but it might be too 
late. A consequence of this lock-in is that there exists two completely distinct inter- 
faces for dealing with discrete pictures vs. continuous pictures. Another lock-in that 
occurred, despite our last-minute efforts to dislodge the error, is interlaced video in 
the national high-definition television standard, HDTV. 

And another set of ideas are the self-criticisms: How many of the technological 
developments were mistakes or dead-ends? Or were they? What did we think we 
were doing at any one time? 

A final theme I emphasize is the lack of artists breaking the bonds of the models 
we early technologists defined for digital light and what we might do to inspire them 
to break those bonds. 


Computation and Moore’s Law 


Let me first pay tribute to the ‘fluid’, so to speak, that makes this discussion pos- 
sible. The notion of computation is a new one really, dating essentially from about 


1936. There is the quibble about Babbage and Lady Lovelace, but Ill ignore that pip, 
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much as historians of America skip over An Order of Magnitude Every Five Years 


. 5 7 . X 2 (10x in 5) 
Leif Erickson’s arrival in North America 
about 1000 AD in favour of Columbus in 
1492. There is no continuous intellectual Toy Story 3 
1,000,000,000x 
thread that descends from the earlier 
x n Disney 
events. In 1936 the notion of universal 
Toy Story 2 
computation, effective procedure, or Toy Story 
1,000,000x 
recursive function, etc. was derived by Lucasfilm Pixar 
New Mexico 1,000x 


the likes of Church, Turing, Post, Kleene 1x 
and others. Beginning particularly with 1965 1980 1995 
‘Saint’ Turing, there has been an explo- _ l f 

Figure 1. Moore’s Law in meaningful form. 
sive growth of the notion of computabil- Courtesy of Alvy Ray Smith. 
ity into actual machines that realize the 
universal computation controlled by software to designate which of the infinity of 
computations is to be universally simulated. Digital light is a subset of this world, 
and appeared early in the history of computation (at least as early as 1947 that I 
have been able to ascertain). The background to this discussion is explosive, expo- 
nential growth. 

The fact that the growth from 1936 has been not just firm and steady, but explo- 
sive and revolutionary, is captured by Moore’s so-called Law. By reworking the math 
just a little, one gets a useful and intuitive statement of Moore’s Law: Anything good 
about computation gets better by an order of magnitude every five years. Or ‘10x in 
5’. So, taking 1965 as about the time when computers began being realized with 
integrated circuits (to which Moore is restricted), we have been riding Moore’s Law 
for 45 years. 45 years divides by 5 years into 9 units—9 cranks of the Moore’s Law 
handle—so Moore’s Law predicts a 10° improvement in computers between 1965 and 
2010. That’s an improvement by a factor of a billion (1,000 millions, see fig. 1). Soa 
kilobyte of memory in 1965 would be 10°*10° = 10” bytes today, or 1 petabyte. Pixar 
measures its memory in petabytes today. I tracked the development of computations 
devoted to computer graphics for many years and it did follow Moore’s Law almost 
exactly. I’ll give examples later. Ed Catmull and I used the ‘Law’ for years to make 
business decisions—it was that regular. Importantly for our discussion here, the dig- 
ital revolution has been revolutionary—and is still fully underway. This is the force 
field underlying everything about which I speak. 

I like to say: The computer is the most malleable tool ever invented. It is our 
goal—and my constant challenge to artists, the explorers at the edges of our cul- 


ture—to understand what the medium of computation really is, as opposed to what 
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we technologists (mostly) pulled out of the technology in our mad rush to the digital 
convergence and the first computer-generated major motion picture. 

The amuse-bouche for this section is: ‘There are no Os and 1s in a computer’. 
Some people find this surprising, thinking that computers are somehow based on 
numbers. They are not, anymore than DNA is based on words from the alphabet 
A, C, G and T. Sir Charles Sherrington’s famous 1942 quote about the human cor- 
tex could as well describe a computer’s changing patterns of energy: ‘an enchanted 
loom where millions of flashing shuttles weave a dissolving pattern.’ Back in the 
early days about all we could interpret from the crude computational power was 


numbers. In these marvellous times, digital light is an exciting interpretation. 


Taxonomy 


I have found the following three distinctions—stated as tenets—of high usefulness in 


understanding picturing, or digital light, and problem-solving in that domain. 


1. The continuous is fundamentally distinct from the discrete. 
2. Creative space is fundamentally distinct from display space. 
3. The interactive is fundamentally distinct from the passive. 


These distinctions are orthogonal to one another and dictate a 3D space in which 
technologies of digital light conveniently fall. I will give several examples after dis- 


cussing the tenets and their meaning. 


1. Continuous vs. Discrete 


This distinction may seem obvious, but my experience is that most people, even my 
colleagues, often have not appreciated its full impact. 

There are two branches of digital light corresponding to the distinction. The 
first is geometry-based and the other is sampling-based. I will assume that everyone 
knows geometry and its heroes, such as Pythagoras and Descartes. I suspect far 
fewer know the all-powerful Sampling Theorem—the basis of sampling-based digital 
light technology—and its heroes, such as Shannon (see fig. 2). 

The Sampling Theorem underlies all digital light technologies. I could devote this 
entire paper to its wonders. But it has to suffice here that you understand that it 
exists, is well-understood, and is the glue that holds the two worlds, discrete and 
continuous together. Let me limn the Sampling Theorem in its most basic form: A 
continuous space of an infinity of points can be accurately represented by a discrete 
space of countable points IF the points (pixels) are chosen correctly. And, further- 


more, one can get from the discrete points back to the continuity they represent 
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accurately IF the reconstruction is Geometry Domain 


done correctly, and the Sampling 


Theorem tells how this is done. In other Rendering 


words, IF ONE IS CAREFUL, a discrete 


set of point samples is equivalent to a 


Geometric 


continuous infinity of points. Otherwise Model 


none of our digital displays would work. 


Reconstruction 


We think we are looking at a continuum 


when we watch digital light. To a large 


Conti l 
degree the history of computer graph- ontinuous (analog) 


ics is the learning of how to cross back 
and forth across the discrete/continu- 
ous border defined by the Sampling 
Theorem. It is so fundamentally important that I think it should be taught in every 
school to every child. It defines the modern world. 

The computer animation of Pixar is geometry-based. The sets and characters are 
defined with geometrical elements, assumed to move continuously through time. But 
consider digital photography. There is no geometry at all involved. The ‘real world’ is 
sampled with an array of sensors on a rectilinear grid. 

The two worlds have different terminologies and different software applications. 
An early pair of apps makes the point: Apple’s MacDraw was geometry-based; its 
MacPaint was sampling-based. Today Adobe has Illustrator for 2D geometry and 
Photoshop for 2D sampling. The interfaces of the two worlds are completely differ- 
ent. You can’t know how to operate Photoshop by learning how to operate Illustrator. 
(I think this is a ‘locked-in’ error, by the way, and is not necessary.) 

In the early days the two branches held side-by-side conferences. As a new assis- 
tant professor at NYU straight out of graduate school, my first chairman was Herb 
Freeman. He had established the first learned journal in the field, called Journal of 
Computer Graphics and Image Processing. 

The history of ‘computer graphics’ often fails to acknowledge the simultaneous 
development of the sampling side. For example, I’ve often heard that Ivan Sutherland 
was the father of computer graphics, but I can’t see how that could be true. He came 
along a decade or more after the earliest computer picturings that I know of—on a 
Williams tube (32 x 32 memory array of dots on a CRT type tube)—which is sampled, 
not geometric, and appeared in the late 1940s and early 1950s. Ivan did important 
early work in the 1960s in interactive geometry-based digital light, but there was 


Image 


or Sprite 


Figure 2. Geometry vs. sampling. Courtesy of Alvy Ray Smith. 
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simultaneous work going on in sampled imaging, such as the Tiros and Nimbus 
weather satellite imagery. 

That people often confuse the two spaces is usually evident around the notion of 
a pixel. Many, many think that a pixel is a little geometric square, but it is not! Itisa 
sample at a discrete point. It has no geometry (unless one wants to think of the point 
where the sample is located as its geometry, or alternatively that the reconstruction 
filter required by the Sampling Theorem supplies the geometry). 

The two worlds are often mixed, and deliberately so. For example, classic film 
cinema has a 2D spatial continuum (the film frame) and a sampled (24 frames per 
second) temporal dimension. Another way the worlds are mixed is via rendering 
techniques such as texture mapping, where a sampled image is mapped to a geo- 
metric surface. 

Why is there such a confusion? Because the geometric world cannot be seen. It 
is abstract. The only way to see geometry is to render it into a sampled image that 
can be displayed. So both approaches lead to pixels. Which brings me to the second 


major distinction. 


2. Creative Space vs. Display Space 


One might think that this is just a variation on the Continuous vs. Discrete distinc- 
tion, but the case is more subtle than that. I will start by explaining where the dis- 
tinction comes from and then extend it to digital light in general. 

Classic 3D computer animation is done as follows: A 3D stage set is modelled in 
abstract geometry, with cones, spheres, cylinders, lines, points, splines, patches, etc. 
Articulated models serving as the characters are similarly modelled. Animation is 
provided by controlling the continuous paths along which parts of the models move. 
As already mentioned, such a scene cannot be seen, so an abstract camera is mod- 
elled too. It is positioned in the 3D space and aimed at the portion of the scene which 
one wishes to see. Then what the camera sees is rendered into a frame. This means 
exactly that the geometric scene is sampled, according to the Sampling Theorem, 
into a rectilinear grid of samples, called pixels. In summary, the creation is done 
in creative space, which is continuous and abstract, but viewing is done in display 
space, which is sampled and discrete, dictated by the display technology being used. 

What is interesting is that there is no reason that once we have distinguished 
creative space from display space in 3D computer graphics, we don’t apply the dis- 
tinction more generally to all digital light technologies. I think this is profound. It 


goes against what we’ve always thought to be the case: That creative space and 
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display space are the same thing. In the old days it was true. Consider a painter and 
his canvas: creative space IS display space. Consider sculpting a block of marble: 
creative space is display space. Consider Photoshop: Creative space is display space 
(and one of the reasons I think the wrong idea has been locked into this pervasive 
program). Why not have 2D imaging apps that have a camera viewing an infinite dis- 
crete creative space? Why not indeed! 

Many of the problems I’ve encountered in digital light boil down to confusion 
about which space one is in, continuous vs. sampled, or creative vs. display. Once 
the distinctions are called, then the problems usually resolve themselves. There is 
a well-known theory, by the way, that instructs us how to convert from continuous 
space to the discrete space and back. It is a direct consequence of the Sampling 
Theorem, but this is not the place to go into it. 

A case in point is the Visible Human Project at the National Library of Medicine. 
I was a member of the board of the Library when the Visible Human was proposed. 
The board was stumped on how to approach the problem of digitizing a human body 
completely in 3D (with 3D samples that is) and extracting models from the data with 
which to train surgeons. I was able to convince them that the two problems were in 
different spaces. The digitization problem was a sampling problem. Solve it first, 
then use sampling theory to derive geometric models from the samples as a second 
step. This is exactly what we proceeded to do and surgeons are now training on 
simulators derived from the sampled human male and female. 

A case where we failed was the national high-definition TV specification. The 
problem here was that we computer people came late to the game. The power and 
the money—always the controlling forces in Washington—had conspired to define 
HDTV as a digitized form of the old-fashioned analogue TV standard of the 1950s. 
In a last-minute battle, we managed to include progressive scanning in the stan- 
dard but were unsuccessful in ridding it of the old interlaced scanning format of the 
1950s. Progressive scanning is what every computer monitor uses, one line after 
another is painted down the screen in quick succession. The old interlaced idea—one 
that no longer made any sense in the digital world—was to paint all the even lines 
first then return to the top and paint all the odd lines. The ugly interlaced standard 


was locked into the standard, and has cost the consumer dearly. 


3. Interactive vs. Passive 


Here is the gist of the third tenet. It is obvious that interactivity is distinct from 


passivity, but as opposed to the other two distinctions I’ve drawn—continuous vs. 
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discrete, and creative vs. display spaces—the interactivity vs. passivity dimension is 
really an axis, not a hard distinction. That is, there are varieties of interaction, from 
none at all (passivity) to full ‘thumb candy’ exercised by adolescent—and probably 
not so adolescent—boys with their point-of-view killer video games. The purpose of 
this tenet is to include under the tent of ‘digital light’ such things as aircraft simula- 
tors (games for big boys), video games (for little boys), caves (fully immersive envi- 
ronments), paint programs, even windows-based UI’s on ordinary computers. And 
even a PowerPoint slide show creation—which is highly interactive—and its quite 
dumb display. The sequence of clicks in a PowerPoint display is interactive, not pas- 
sive. Furthermore, it is discrete. So continuous vs. discrete applies to interactivity 
as well. Despite the different levels of interactivity that are possible in digital light 
technologies, I am going to use the simple distinction interactive vs. passive as the 


basis of a taxonomy in this paper. 


4. Examples 


The three tenets define Display space examples 


three orthogonal axes 


Film photo 2D 2 continuous passive 
which divide up a taxonomi- Digital photo 2D 2 discrete passive 
cal space into eight regions. Device interface 2D 2 discrete interactive 
It is simple to drop all forms ; h 2 continuous : 

Classic movie (2+1)D Fidi t passive 
of digital (or analogue) light ising 
experiences into the appro- Digital movie (2+1)D 3 discrete passive 

` A Video game (2+1)D 3 discrete interactive 
priate region, as shown by 

Volumetric image 3D 3 discrete assive 
these examples. Note that I 2 E 

Volumetric movie (3+1)D 4 discrete passive 


actually catalogue via each 


dimension of each technol- 
ogy, and distinguish spatial 
from temporal dimensions Creative space examples 


by using the + conven- Oil painting 2D 2 continuous interactive 
tion—for example, (2+1)D Paint program 2D 2 discrete interactive 
means 2 spatial and 1 tem- Drawing program 2D 2 continuous interactive 
poral dimensions. Illustrator file 2D 2 continuous passive 

3D modelling program 3D 3 continuous interactive 
5. What About Displays? Marble sculpting 3D 3 continuous interactive 

Volumetric sculpting 3D 3 discrete interactive 
I do not use display device 

Animation program (3+1)D 4 continuous interactive 


differences as part of the 
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taxonomy. For example, whether display is via transmission or via reflection is not 
used, or whether the display uses additive colour or subtractive colour. Why is that? 

We certainly spent years mastering different display devices, but in retrospect 
that was part of the old, pre-convergence era. We had to master film, video and ink 
on paper as truly different display media. We had calligraphic displays and raster 
displays. We mastered the technical details of each display, its colorimetry, its stan- 
dards, its nonlinearities, etc. For example, I spent years mastering the art of reading 
and writing bits with colour lasers to and from trilayer Kodak colour movie film. But 
the digital convergence to digital light has made all those distinctions irrelevant—or 
at least of greatly diminished importance—in a way that I will now try to explain. 

Part of the idea of convergence to a single medium, namely the computational 
medium, was the relegation of display device quirkiness to the edges of our world. 
The creative world is linear, whether it be discrete or continuous. We could so define 
this because it was abstract. We always knew that once the ‘rubber met the road’— 
the actual display in the real world of our creations—the nastiness of the actual 
devices would have to be taken into account. Real display devices are never linear, 
and they are seldom nonlinear in the same way. As usual we did not originally know 
that we were dividing the world this way—originally we often confused the details of 
display with details of the underlying creation. 

So here is the trick: the actual details of a particular display device technology 
is relegated in the computer-mediated pipeline to the last possible position in the 
pipeline, in what’s typically called a ‘device driver’. Device drivers are difficult to 
write and there is a different one for every monitor on every phone, on every game, 
on every computer, and a different one for every projector, and for every plotter and 
printer, whether to ink on paper or film or photographic paper. It is the device driver 
that handles the transformation (nonlinear in general) from the inner linear creative 
world to the actual hardware device. Here is where the notion of subtractive vs. 
additive, aspect ratio, gamma correction, etc. is handled. This is where the gamut 
of the colours of the inner world is transformed into the gamut of the display device. 
Here is where each manufacturer’s technical choices are accounted for. 

The same thing happens on input (or should): the nonlinearities of the input scan- 
ner, for example, are straightened out before the bits are passed to the main pipeline. 

Note that display devices are not to be confused with display space. Each display 
space can map to an infinity of display devices (of course, there isn’t an infinity, but 
the number and variety is always increasing). It is an important trick that we sever 
the creation of images, which should work universally, from the nasty details of dis- 


play devices, which are specific, local, and vary with manufacturing advances. (An 
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aside: An underappreciated accomplishment of Microsoft, who does not make hard- 
ware (games excepted), is that its software works on thousands of different devices 
made by hundreds of different manufacturers.) 

It is a mistake, in this view, that nonlinearity, such as gamma correction, be 
included in creative space, whether discrete or continuous. But, unfortunately, the 
notion is locked in as an artefact in some modern very popular applications. 

There is a problem in the world of display devices that should bother us more 
than it has so far. It has to do with the misrepresentation of the colours of one world 
by the colours of the displays we choose to use. I have in mind the real world and the 
human colour gamut therein—all the colours that most humans can see in the world. 
No display device can replicate this gamut, so whenever the real world is repre- 
sented with digital light, its gamut has been reduced. Another example: Google has 
announced a project of putting famous art galleries online. This sounds great, but I 
am amazed that the colour issue is never raised. The colours of the oils and acrylics 
and watercolours, etc. in the galleries are not correctly reproduced by the digital 
light devices. The surprise is that we humans are so tolerant of the colour loss inher- 


ent in digital light as realized so far. 


Genealogy 


I am a scholarly genealogist, in the human family history meaning of that term. In 
its strictest interpretation it is the tracking of the path the genes take. I shall use it 
here to mean the tracking of the germ of an idea. 

The ‘families’ I shall track are those defined by the taxonomy. Plus hardware 
and devices. This is not the way it happened, of course. The taxonomy is my current 
understanding of what we did, mapped back onto our actual activities. 

I have organized this paper along a timeline defined by my career through sev- 
eral centres of graphics development. This naturally cannot encompass the entire 
history of development of digital light, but the centres I mention (in this order)— 
Xerox PARC (Palo Alto Research Center), University of Utah, NYIT (New York 
Institute of Technology), Lucasfilm and Pixar—happen to be the places that did much 
of the development themselves, and are hence representative of the whole field. 

Because there were so many developments I shall skim over many of them, per- 
haps simply listing them, and concentrate on the profound turning points. There 
were many dead-ends pursued during the time period I concentrate on (1965-95). We 
spun our wheels in many areas (2D geometric inbetweening). We devoted too much 


time to specific displays (analogue video). We devoted too much time to speeding-up 
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programs when a simple wait for Moore’s Law to crank would have done the job (tint 
fill optimization). We often didn’t really understand what we were doing, but in the 
context of knowing exactly what we were headed for (the first completely computer- 
generated feature film). We worked under models inspired by ‘reality’, but not bound 
by it. Our theoretical understanding of what we were doing was often fuzzy and only 
now being understood (for example, premultiplied alpha). We’ve already allowed 
several bad ideas to be locked in (rectilinear images, interlaced TV, two conflict- 
ing UI models for pictures). Many of the things we started were never finished and 
just begging to be done (what ever happened to computer painting?). There are still 
some very tough problems to solve (live action with human actors). I will attempt to 


revisit some of these along the way. 


1. New Mexico, Our Outback (Moore’s Law 1X, 1965) 


I start the Moore’s Law ‘clock’ at about 1965 for two reasons: (1) Gordon Moore 
himself came up with the ‘Law’ for semiconductor chips which first began to be 
used about then. (2) I made my first computer graphic in Las Cruces, New Mexico, 
then (actually 1964), a plot of a 2D equiangular spiral for a Nimbus weather satellite 
antenna (2D, continuous, passive). 

There were other early computer graphics happening about this time other 
places: Ivan Sutherland’s Sketchpad (2D, continuous, interactive, with discrete dis- 
play!) in 1962-63; DAC-1 (2D/3D, continuous, interactive) in 1959-64 at General 
Motors; and Bezier patches (3D, continuous, passive). On the image processing front 
(2D, discrete, passive), there were systems for processing early weather satellite 
imagery. For example, that from the TIROS, the first weather satellite in 1960. 

In relation to New Mexico: there was the development of the Bresenham algo- 
rithm in 1965, considered the earliest ‘rendering’ algorithm. It converted straight 
geometric lines to point samples (pixels) on a grid (2D, continuous, passive; ren- 
dered into 2D, discrete, passive). The surprise to me was to learn, years later, that 
Bresenham and I came from the same hometown, Clovis, New Mexico. Our moms 
knew each other. 

The display devices were varied at this time: I used a line plotter for the antenna, 
Bresenham a point plotter, Sketchpad a raster display (which surprised me to learn) 
but calligraphic output (line plotter), DAC-1 a scanning raster display (it would scan 
a pencil drawing and represent it internally as a set of cubic polynomials—i.e., a 
spline); commands could push the design into 3D, output to a milling machine or 


plotter. Bezier patches were defined mathematically and may have been used to 
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drive a Renault automobile surface milling machine, which would have been there- 
fore the ‘display’. 

One of the earliest computer graphics animation that I’ve heard of was a wine 
bottle pouring wine into a wine glass, on the spots of a Williams tube. I was told 
about this animation by an old IBM engineer who had witnessed it in 1952. The 
Williams tube was the 32 x 32 memory device developed originally for the early 
Manchester computer in the late 1940s. So the graphics was quite crude. And it was 
discrete! I have still pictures from the Manchester Williams tube from 1947, the ear- 
liest graphics that I know. 

Pinscreen animation (Alexandre Alexeieff and wife Claire Parker, and Norman 
McLaren) at the National Film Board of Canada in 1960s and 1970s was discrete. 
(Their first was in 1933!). Orson Welles’ The Trial in 1962 featured a pinscreen title 
sequence (not animated). But Hunger, by Peter Foldes (1974), also at the NFB, was 
geometric and continuous. (It was nominated for an Academy Award.) 

Edward Zajec and Michael Noll made early computer films (early 60s). Chuck 
Csuri made early computer art (MOMA purchased one for its permanent collec- 
tion, in 1968). Ken Knowlton also contributed in the 1960s. The book Cybernetic 
Serendipity brought a lot of early computer art into focus for me, in about 1968. 

Spacewars, perhaps the first widespread ‘video’ game, was invented at 
MIT in 1961. 

Herb Freeman, who was to become my chairman at NYU in 1969-74, had started 
in computer graphics in 1961, and had students working on quadric surfaces, for 
example, when I was an assistant professor working for him. One of his students 
(Loutrel, 1966) solved the hidden-line problem, which was big at the time, and has 
simply disappeared as a problem now. He and Azriel Rosenfeld started, in 1970, the 
Journal of Computer Graphics and Image Processing, later the Journal of Computer 
Vision, Graphics, and Image Processing. The point is that this division of the digital 
light world existed early on. 

Lee Harrison’s Scanimate, an attempt at video animation, came to my careful 
attention early on; he and Dick Shoup (see PARC, below) were friends. As did Ron 
Baecker’s Genesys animation system (which I saw in the late 1960s at NYU; later 
Ron and I would briefly share a house in Redwood City, California). 


A Taxonomy and Genealogy of Digital Light-Based Technologies 


2. Xerox PARC (10X, 1970-75) 


It is well-known that the personal computer, as we now understand it, was being 
invented at PARC during this time. For digital light in particular, the following ideas 
were being pursued: 

Colour paint program (SuperPaint, by Dick Shoup), 2D, discrete, interactive, cre- 
ative space was display space. The colourmap made 8 bits capable of producing 16 
mega colours (but only 256 at a time). 

Colour transforms (RGB to HSV, by me). This was really a cheap trick. I spent 
years explaining that, despite its names, such as ‘hue’, it had nothing truly to do with 
human colour perception, other than a chance resemblance. The exercise did excite 
a long scholarly interest in me for colour science. 

Window-based UI, 2D, discrete, monochrome, interactive, creative space differ- 
ent from display space (the underlying creative metaphor being the desktop covered 
with papers). 

Antialiased geometry (Dick Shoup and Bob Flegal), both passive and interac- 
tive, 2D, monochrome, creative space was geometric, so continuous, display space 
was discrete. 

In 2D geometry space: a text and graphics description language (i.e., a 2D model- 
ling language) was in development with John Warnock and Chuck Geschke, eventu- 
ally leading to PostScript and then to Adobe. 

Interestingly, William Newman, who with Bob Sproull wrote the first computer 
graphics text, was there. I say interesting because we colour guys essentially lost 
out to the black-and-white guys in Xerox’s infamous decision to not pursue the per- 
sonal computer and to pursue black-and-white rather than colour. Even more inter- 
esting, William Newman was son of Max Newman who with Turing programmed the 
first computer in Manchester in the 1940s. 

I do not include in my essay analogue computer art. For example, John Whitney 
(Sr.) used bombsight ‘computers’ to drive a flying spot scanner onto film to make 


early interesting films in the 1960s. Iam strict about observing the ‘digital’ attribute. 


3. University of Utah (1X-10X, 1965-75) 


Although I never worked at the University of Utah or at the Evans & Sutherland 
Corporation, I feel justified in including them as if I did. This is because I made 
an early pilgrimage to Salt Lake City that figured into my future, and that future 
included many people who came from the University of Utah hotbed and much 
equipment (and some people too) from the Evans & Sutherland Corporation. Most 
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important of these, of course, was Ed Catmull, who cofounded Pixar with me. Here 
is the list of people: 

CHRISTINE BARTON, MaALcoLM BLANCHARD, JIM BLINN, ED CaTMULL, JiM CLarK, Elaine 
Cohen, EpHraiM CoHEN, Steve Coons, Frank Crow, Gary Demos, Dave Evans, Henry 
Fuchs, Henri Gouraud, Jim Kajiya, Alan Kay, Martin Newell, Fred Parke, Bui-Tuong 
Phong, Rich Riesenfeld, GARLAND STERN, Tom Stockham, Ivan Sutherland, John 
Warnock, Lance WituiaMs. (The names in small caps were with me at NYIT. The 
names in italics directly affected my career in other ways: Alan Kay helped me get 
hired at PARC; Martin Newell helped me get hired at NYIT (it is his famous teapot 
that became the icon of computer graphics); John Warnock tried several times to 
either buy my technology or hire me at Adobe; Gary Demos was my archrival (with 
John Whitney Jr. at Information International Inc. and then Digital Productions) and 
then later an admired collaborator with me in the HDTV battles in Washington, 
D.C.; Jim Kajiya and Jim Blinn worked with me at Microsoft, Jim being the only other 
Graphics Fellow there; and I roomed with Jim Blinn during my short stay at the Jet 
Propulsion Lab of Cal Tech between NYIT and Lucasfilm, while working on the Carl 
Sagan Cosmos television series.) 

Utah was an early powerhouse of 3D geometry-based computer graphics. 
Amazingly they at first did not have a real-time output device. They would generate 
files of pixels from their rendering algorithms and submit these to a person whose 
job was to convert each file into a photograph, done using a flying spot monitor and 
a camera devoted to it. So these people worked in creative space entirely (except for 
the final files, rendered to display space). 

Among the many accomplishments of this group are these (where I use ‘shaping’ 
and ‘shading’ to describe two aspects of modelling in creative space, one defining 
geometry, the other appearance): 

In shaping: patches, B-splines, patch subdivision. In shading: Gouraud shading, 
Phong shading, texture mapping. In rendering (that is, the conversion from continu- 
ous creative geometry to discrete display samples): hidden surface elimination algo- 
rithms in continuous space, Z buffer hidden surface elimination in discrete space. 
In 3D geometry-based (continuous) animation: Ed Catmull digitized his own hand, 
covered the mesh so created with simply shaded polygons, and flew a virtual camera 
up the fingers which moved. This short test eventually made it into the movie Future 
World. And Fred Parke animated a face and lip-synced the dialog. 

Antialiasing: Tom Stockham, in the digital audio world (he got famous for analyz- 
ing the 18-minute gap in Richard Nixon’s Watergate tapes), taught antialiasing to 
Utah students. Frank Crow wrote a thesis on it applied to computer graphics, but all 
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the students imbibed the theory and brought it with them to NYIT. (It is worth not- 
ing that digital audio was part of our world at NYIT and Lucasfilm. The problems it 
was solving were the same as the ones we were solving visually.) 

Hardware: Jim Kajiya helped design the first RAM-based framebuffer for the 
Evans & Sutherland (E&S) Corporation. NYIT was to be the first customer of this 
crucial device (now known as a graphics card). Gary Demos was a logic designer 
for E&S hardware (I first noticed him because only his logic diagram was projected 


from a 3D model!). 


4. New York Tech (100X, 1975-80) 


The number of things pursued at NYIT was vast and varied. I list here some of them 
which seem most pertinent to the genealogy of digital light, emphasizing those I 
believe were most profound and lasting. But first let me mention the context for 
these developments. 

The NYIT campus was heavenly, located on grand estates on the fabulous North 
Shore of Long Island, the location of F. Scott Fitzgerald’s The Great Gatsby (1925). 
It was owned by our first patron, Dr. Alexander Schure, or Uncle Alex as we called 
him. There was a full animation studio on site, of the classic 2D cel animation vari- 
ety, with some of the early animators, like ‘Johnny Gent’, who animated for Max 
Fleischer. Alex intended for us to make him money by (he thought) getting rid of the 
labour in the animation process. For our purposes, we were funded better than any- 
body else in the world, and thereby got access to RGB before anybody else (the first 
RGB framebuffer cost Uncle Alex the equivalent of today’s $1,000,000, for a (merely) 
video resolution picture memory of 24 bits per pixel). NYIT became a Mecca for 
those interested in computer graphics. We got a master’s lesson in how animation 
was actually done, an education in logistics as much as the craft itself. Everything 
we did was aimed at animation, broadly interpreted (so including TV commercials, 
spots and video animation). New York City was 8 miles away, so I became immersed 
in the nascent video art world there. Artist Ed Emshwiller lived on Long Island and 
eventually found us, becoming my mentor; we created Sunstone together, my proud- 
est artistic achievement (and now included in MOMA’s permanent collection). 

The hardware and software context cannot be ignored although strictly speaking 
it had nothing to do with the genealogy of ideas. We had the first VAX minicomputer. 
We had the first RGB framebuffers. We had a full video editing facility in a nearby 
mansion. We had some of the earliest access to Unix and its programming language 


C. We usually got what we asked for. We also had a digital audio team. 
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Back to the genealogy: 


2D display space: RGB everything, paint3 (for background art in 2D 
animations), tint till (for opaquing in 2D simulation of cel animation), 
scan’n’paint (the 2D simulation of cel animation), real-time video frame- 
grabbing (part of the scan’n’paint system). And the most profound of 
all: the alpha channel, and matting. (I contend that the profundity of 
alpha is still not appreciated.) A frame language, utilizing the Unix 
shell, was designed for creating arbitrary combinations of 2D func- 
tions, used mostly for complex matting sequences, but also for reposi- 
tioning, shadowing, sharpening, reversal, colourmapping, etc. 100s of 
art pieces created by various members of the team. Ed and I developed 
the ‘2-pass’ scanline-oriented transformation process (which actually 
required reconstruction of pixels into continuous space, transforma- 


tion, then resampling into display space). 


2D continuous space: tilings, splines, Tween (Ed’s attempt to do cel 
animation geometrically, which essentially failed after major effort; 3D 


geometric animation is easier than 2D)). 


(2+1)D discrete space (all 3 dimensions discrete): Measure for Measure, 
done almost completely with scan’n’paint. Dissolves, wipes, and fades 
(dozens of them). A video recording language was developed to drive 
frame-by-frame editing (i.e., the first frame-accurate digital video edi- 
tor) of such pieces as Measure for Measure, and many art pieces and 
demo pieces generated by the team. Some of the earliest TV commer- 
cials, spots, logos, and rock videos were created this way. Sunstone 


was mostly generated this way. 


(2+1)D space, 2 continuous (space), 1 discrete (time): We generated 
continuous, analogue video masters from the dissolves, wipes, and 
fades mentioned above, which were then used to control real-time ana- 
logue videotape dissolves, wipes, and fades. David DiFrancesco mas- 


tered film recording from digital images. 


3D continuous space: 3D ‘soids’ (ellipsoids), patches, modelling lan- 
guage (e.g., I was able to model skeleton hands, skinned by Ed’s Utah 
hand for a Siggraph cover). Ed solved the hidden surface problem of 
geometric entities at the pixel level—that is, if the world was divided 
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into little squares, each of which would map to a single pixel at render- 
ing time, his algorithm solved the hidden surface problem in each such 
little square (it was working with him on this when we came up with 


the alpha channel concept). 


(3+1)D continuous space: I created Texas (Texture Applying System) 
which modelled rectangular geometric flats in arbitrary relative 
position in 3-space, with virtual cameras for display, and with ani- 
mation controls. Each flat had an image mapped to it—that is, was 
texture-mapped. 


Rendering: Lance developed shadow maps. Texas used the 2-pass algo- 
rithm to render transformed, animated flats from the texture given 
for each. Jim Blinn visited for several months, and did bump mapping 


while he was there. 


Display devices mastered: digital film recorder, NTSC television, PAL 
television, video tape recorders, RGB broadcast-quality monitors, rho- 
dium coated videodisc recorder, single-shot camera from monitor, and 
all the gamma nonlinearities and colour gamuts involved. Also, text 
monitors, E&S Picture System calligraphic displays, Tektronix green- 


on-green text and line-drawing displays. 


Also, many tablets devices were tested, including pressure-sensitive ones (which all 


failed). The mouse was not interesting to us. 


5. Lucasfilm (1000X, 1980-85) 


The context at Lucasfilm was of course the movie business, and for us the special 
effects house, Industrial Light & Magic. George Lucas had hired us principally (only, 
it turned out) to build three instruments for him: a digital optical printer (which 
we realized with the Pixar Image Computer and laser film reader/writer), a digital 
video editor (which we realized as EditDroid), and a digital audio synthesizer (which 
we realized as SoundDroid). It came as a surprise to us that George did not seem to 
want us in his movies, but not until I had assembled a world-class team to do just 
that. We had our first go at the big screen in Star Trek II: The Wrath of Khan (via 
Paramount, not Lucasfilm). THEN George put us in The Return of the Jedi. And then 
Steven Spielberg, George’s friend put us in The Young Sherlock Holmes. 

One of the biggest events for our group (that would become Pixar) was the hir- 


ing of John Lasseter as our animator, and my directing him in his first piece, The 
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Adventures of André & Wally B. A surprise was to discover that George did not believe 
we could do animation. Another big event for the Graphics Group was the deal made 
with Disney to digitize their cel animation process. We, of course, had done the first 
steps already, at NYIT with scan’n’paint. The result was CAPS (Computer Animation 
Production System), a very successful project that bonded us with Disney, finally. 
Then George and Marcia Lucas divorced, leaving him without enough money for 


us. At that point Ed and I decided to spinout our group as a separate company, Pixar. 


We spent a lot of time finding funding for the new venture. 


At the very beginning of the Lucasfilm era, we emphasized that no code was to 
be written that did not correctly antialias. I created ‘No Jaggies’ T-shirts to make 


this requirement known, internally and externally. 


Now back to the genealogy: 


It should also be mentioned that Ed Catmull was asked by George to start a games 
group, which he did. Although this was not our main interest, it was definitely a 


2D display space: RGBA everything, paint4, premultiplied alpha and 


matte algebra, advanced 2-pass algorithms. 


2D continuous space: tapered splines (which was actually a rendering 


technique from continuous into discrete space). 


(2+1)D discrete space (all 3 dimensions discrete): digital video editing 
project (not aimed at broadcast video, but at cinema), CAPS. 


3D and (3+1)D continuous space: modelling language, articulated vari- 
ables, fractals, graftals, particle systems, matrix package (for splines, 
for scene transformations, for transformations at joints of articulated 
models), view package (for virtual camera control of perspective and 


other view parameters), motion blur control. 


Rendering: motion blur realization, materials properties, universal 
solution for antialiasing, Reyes = Renders Everything You Ever Saw, 


micropolygons. 


Display device mastered: Laser to trilayer film printer (built by David 


DiFrancesco). Also a laser reader of film. 


strong move in the interactive world. 
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6. Pixar (10,000X, 1985-90) 


The context at Pixar was, at first, hardware and keeping our company alive until 
Moore’s Law cranked another turn or two. The Pixar Image Computer was the 
world’s fastest discrete 2D machine for a short time. We did image processing on it. 
Textured terrains. Hydrodynamic flows. Material stress analysis. Medical imaging. 
The animation group continued to produce better and better shorts, as Moore’s Law 
gave us power and more features were packed into the software. CAPS was success- 
ful (Rescuers Down Under in 1990 was arguably the first completely computer-gen- 
erated movie, except the artwork wasn’t done digitally). Then in 1991 Disney came to 
us and proposed that they produce a computer-animated film and that we produce it. 
That was, of course, Toy Story (1995). The story of Pixar since then has been one of 
movie after movie, all successful. Many techniques for handling ever more complex 
productions were created. Growing pains for a company, from 200 to 1200 persons. 
Building rendering farms of very increasing computational power and memory size. 
I departed in 1992 in a spinoff, Altamira, which I then sold to Microsoft. I departed 
when I knew that John Lasseter would be comfortable in working with Disney (who 
had fired him) and that Toy Story was a go. Altamira was devoted to 2D and (2+1)D 
discrete space, an image composition application featuring sprites—i.e., non-recti- 
linear images. 

Toy Story 2 was released in 1999. I performed tests with my friends at Pixar that 
compared frames from Toy Story to frames from Toy Story 2, but both executed on 
the same hardware. During this crank of Moore’s Law (almost) the number of poly- 
gons doubled approximately (from 3-17 megapolys to 4-39 megapolys per frame), 
but total rendering time increased by approximately Moore’s Law’s prediction. So as 
Moore’s Law ‘gave us’ an order of magnitude more computation power, the complex- 
ity of each frame increased accordingly. The same goes on now. 

Back to the genealogy: 


2D display space: All software for the Pixar Image Computer, sprites, 
IceMan language. 


(2+1)D discrete space (all 3 dimensions discrete): CAPS development. 


3D discrete space (all three spatial): volumetric imaging (I become 
trustee at the National Library of Medicine and help create the 


Visible Human). 
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3D and (3+1)D continuous space: modelling and animation improve- 
ments, much of it to adapt to the needs of the animators, lighting con- 


trol, ever increasing lighting model detail, fur, hair, cloth, skin, etc. 


Rendering: the Renderman shading language (perhaps the most pro- 


found development at Pixar after solution of universal antialiasing). 


7. Now and the Future (1,000,000,000X, 2010-20) 


Context: Toy Story 3 was released in 2010, when Moore’s Law hit 1 million X. There 
was enough horsepower to generate it in ‘3D’ (which, of course, is not 3D creative 
space, but 3D display space). This means the movie was computed twice, once for 
each eye. Importantly, in 2009 The Strange Case of Benjamin Button was released, 
featuring the first convincingly rendered human actor, ‘driven’ by Brad Pitt. 

We are not proposing that the computer do the acting. We haven’t a clue how to 
do that. But we are proposing that the computer will convincingly render the avatar 
of the actor. That was proved by Benjamin Button. I looked at it eagle-eyed and just 
could not tell what parts were computer generated. 

Actors and animators are the same kind of talent. Pixar hires animators by how 
well they act. The difference is that actors animate their own bodies, convincing 
us that those bodies belong to someone completely different than the actor, and to 
someone different for each role. An animator convinces us that a stack of geometric 
polygons is alive, and conscious, and feeling and thinking. The difference so far has 
been the quality of the avatar. Pixar has elected to keep the avatar cartoonish. Other 
producers have made it monstrous, and others have made them animals. These are 


all tricks for avoiding the very difficult task of creating a convincing human. 
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CHAPTER 2 


Coherent Light from Projectors 
to Fibre Optics 


Sean Cubitt 


Recent influential accounts of technology recount the history of technical develop- 
ment as a tale of components assembled and reassembled in increasingly complex 
orders, overdetermined by larger systems of energy production, manufacture and 
transport that shape them, to the point of autonomy from human invention (for exam- 
ple, see Stiegler 1998: 76-78). This kind of account is less true of modern visual tech- 
nologies, at least until these underwent a major change in the nineteenth century. 
The archaeological record shows that the light of flames was the only source of noc- 
turnal illumination up to and including the invention of the gas mantle in the 1790s; 
and that hardly any major changes marked the history of lighting by oil lamps (in 
warmer) and tallow or wax candles and rushlights (in colder climates) over a period 
of millennia. It was only with the invention of the light bulb, and even more impor- 
tantly Edison’s central electrical power generation system, that the modern system 
of visual technologies began to form the kind of system envisaged by Stiegler. In 
fact, electric light was not the only visual technology system to come into existence 
in the late nineteenth century: cinema, redeploying older entertainment systems 
and optical devices together with new road and rail transportation, would produce 
a complex new method for the distribution and display of visual arts. More recently, 


television must be seen technically as the offspring of the pre-existing systems of 
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wire telegraphy and radio transmission. Yet, later still, laser and fibre optics would 
bring a third major new system into existence. 

Flame produces soot as a by-product, and, in the large quantities used for public 
entertainments, combustion lighting tends to produce a lot of heat that uses up the 
available oxygen and encourages swooning. Heat is an entropic quality in electrical 
as in mechanical systems: waste energy, with no usable formal properties, radiates 
and dissipates. This is also a typical quality of light itself in the form of flame: as the 
heat vaporises molecules of the fuel, and by-products of combustion fall away, free 
radicals are excited sufficiently to shed excess energy in the form of photons. The 
colour of flame is a function of the kinds of material vaporised, a chemistry which 
remained ill-understood until very recent times, even after the notably late inven- 
tion of photometry and colorimetry (Johnston 2001), part of the rush of invention in 
the nineteenth century (O’Dea 1958; Schivelbusch 1988; Bowers 1998). The latter 
seems to have been led by the rise of mass spectacle, as much as by the require- 
ments of extending the working day in manufacturing, the former seen especially 
in the new department stores which also hastened the development of plate glass, 
and advertising, especially in the extremely popular World’s Fairs (Benedict 1983; 
Geppert 2010). At the time, the expenditure of waste energy must have seemed at 
the time an integral part of the spectacle of modernity. In the twentieth century, 
however, the increasingly integrated systems of electrical power and electric light 
gradually brought this waste under the managerial ideology of efficiency. In this 
process other visual technologies took a lead, starting with the cinema. 

While it is important to understand the originality of these changes, it is also 
important to understand their continuities with older technologies. This chapter 
develops an argument that certain technologies, especially glass and mirrors, have 
been and remain key to both analogue and digital management of artificial light, 
while also arguing that visual media have played a central role in the evolutions 
of related technologies such as glass and mirrors. At the same time, however, the 
surprisingly short history of electrical light allows us a privileged view of how a new 
invention rapidly installs itself as a new system, with all that implies for standardi- 
sation and bias towards innovation—improved efficiencies and applications—rather 


than invention—the more startling emergence of wholly new techniques. 


Projectionists 


Although its roots lie in far older visual displays, such as the magic lantern, it is 


only with the Lumière Brothers’ first public display at the Salon Indien in 1896, that 
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cinema emerges as the first real employer of light as a projection medium. Not long 
afterwards, projected and reflected light was largely overtaken as a general display 
medium for public space. The Moore tubes of 1902 and the first neon tubes from 
about 1910 followed the use of arrays of bulbs as the medium of choice for display. 
Yet, projection did retain some place in public space in the form of arc-lamp based 
searchlights. These were displayed at World’s Fairs from the 1880s, but their exten- 
sive use in military applications would only start with the development of aerial war- 
fare in the 1930s and during World War II. Cinema projection provided the test bed 
for these new forms of organised light. The heart of the projection system is the 
lamp. For a considerable period, well into the 1960s and in some cinemas for far 
longer, tending the lamp was a full time occupation, since the illumination source 
forcing light through the tiny 35mm aperture onto theatrical screens metres across 
was a carbon arc, which burns away in use and has to be adjusted constantly to keep 
the spark brilliant and steady. The xenon arc lamps which came in from the late 
1950s were far easier to manage, but the change-over system of projection—where 
two projectors alternate, with reels of between ten and twenty minutes following 
one another—and the use of rectifiers to increase the voltage sufficiently to power 
the big lamps, meant that projection was a concentrated and technical profession. 
The introduction of ‘cakestands’, holding a whole program (features, trailers, ads, 
music, even cues to open and close curtains and adjust tabs) and of clockwork and 
later digital towers (allowing one or several films to be programmed across multiple 
screens from a single control point) changed much of the old craft, but the principle 
of controlling light carefully and of forcing it through a small area onto the screen 
with the minimum flicker and shudder remains a constant of cinematic projection. 
Piercingly bright for the operator, arc and xenon lamps have to be observed— 
when checking for flicker or wear—through the kind of dense glass plate used by 
welders. But even this much light is a drop in the ocean when it is asked to illu- 
minate a whole wall, metres distant. Every photon is precious. The lamps radiate 
light in all directions. Heat- and light-absorbent linings for the lamp-housing reduce 
scatter, which would interfere with the clarity of the projected image. A condensing 
(parabolic) mirror gathers the radiating light from the rear of the housing and ori- 
ents it towards the business end of the projector, passing it through two condensing 
semi-convex lenses to concentrate the beam before it travels to the gate where the 
filmstrip passes through it. Here the light passes through two compound lenses on 
either side of the filmstrip. The first of these lenses lies between the lamp and the 
filmstrip, focusing the light as close to parallel as possible so that the whole of the 
frame is illuminated with as even a light as possible. The second lens diverges the 
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parallel rays at the precise angle required by the architecture of the cinema theatre, 
refocusing them to illuminate the screen ahead as evenly as possible. 

The light then travels to the screen. The small rectangle of piercingly bright light 
spreads out across as much as fifty metres of space (the ‘throw’) to illuminate the 
screen: from 35 millimetres to, in the case of the Odeon Marble Arch (the biggest 
screen in the UK prior to the 1997 conversion to multiplex), a screen nine metres 
high and twenty-three metres across. This magnification also magnifies any flaws in 
the physical filmstrip. Between the filmstrip and the second lens lies the aperture 
plate, which cuts off the outer edges of the frame, typically ‘noisy’ where the black 
framelines intrude on the image, and commonly the zone where microphone booms 
and other unwanted elements will appear if not masked out. Tabs—black curtains 
drawn up to the edge of the projected image on screen—and the black surrounds 
of the screen itself cut off the outer edges, while the cinema screen is perforated to 
allow sound through and to diminish the effects of air-conditioning on the hanging 
screen. Even with the silvering used to increase the amount of light reflected, the 
screen cannot help bouncing light all around the theatre. The amount of light reach- 
ing the eyes of the audience is thus far less than that emitted in the lamp-house. The 


critical sites for minimising the loss are the lenses. 


Lenses 


A simple lens throws a circular image, one in which, the more the centre is in focus, 
the poorer the focus towards the edges (and vice versa). The simple convex lens— 
like those found in magnifying glasses—produces a number of such aberrations: 
coma (stretching circles into ovals due to focusing different rays from a single object 
at different distances); spherical aberration (corona effects produced by light travel- 
ling different distances through the lens and so being differently refracted); a variety 
of colour fringing effects (‘chromatic aberrations’), caused by different wavelengths 
travelling at different speeds through the glass; and many others, some shared with 
the lenses in human eyes. These aberrations are all the worse in standard window 
glass, which also has a distinct blue-green cast. Projector lenses (like those in use 
in cameras) employ far purer optical glass. Reducing impurities, however, does not 
remove all these imperfections, which stem from the curvature required to expand 
or narrow the focus of light passing through them. To correct them calls for second- 
ary elements to be added to the lens: coatings, and additional glass elements, for 
the most part also lenses, including some glued together and some which use the air 


between elements as an operating part of the system. Lens complexity depends on 
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the aperture in relation to the curvature of the prime lens. A wide-angle lens with 
a large aperture requires more correction, as more of the light passing through the 
distorting outer area of the lens reaches the light-sensitive areas of the frame. Wide- 
angle lenses thus require up to fifteen elements to correct and re-correct the distor- 
tions the prime lens produces, while a long lens with a small aperture may require 
as few as two. 

As well as air, the differences between types of optical glass are critical. Flint 
glass has a high lead content and prismatic characteristics, which neatly match 
and cancel out those of crown glass, a mediaeval invention so called for its shape 
when blown. In the 1880s the Jena-based company Zeiss began manufacturing new 
kinds of glass for scientific instruments, which they then commercialised for the 
photographic market. Paul Rudolph, their master designer, pioneered the practice 
of gluing different elements, and different glass types, together to form compound 
elements. He later combined converging with diverging lenses to reduce or amend 
aberrations. These two techniques proved invaluable, especially in conquering chro- 
matic aberrations. This requires combining a biconvex (positive or converging, with 
two convex surfaces) lens boasting a high refractive index with a biconcave (nega- 
tive or diverging, with two concave surfaces) element which refracts the incoming 
light much less. Zeiss maintained their leading position in lens design after World 
War I by pioneering the use of anti-reflective coatings to improve the quantities of 
light travelling to the filmstrip. These were needed because the increasingly com- 
plex compound lenses absorbed or reflected an increasing proportion of the avail- 
able light away from the film surface (Kingslake 1989). 

Recent glass and plastic lenses include rare minerals like lanthanides and cae- 
sium to control the refractive index and transparency. The interior of the barrel 
housing the glass elements must also be of the deepest black available. In January 
2008 a Rensselaer Polytechnic team announced the blackest material known to sci- 
ence, a forest of carbon nanotubules that absorbed all but 0.045% of visible light 
(Yang et al. 2008). The first use proposed for it was lining light-sensitive scientific 
instruments, and cameras. Such contemporary moves to improve lens and housing 
performance continue to build on century-old design principles. Today, from the 
moment the lens cap comes off to the retinal wall of audiences’ eyes, the aim is to 
retain the greatest density of information possible, even though we know that, ana- 
logue or electronic, the image degrades from the moment light enters the lens of the 
camera to the moment it leaves the lens of a projector. 

The rich history and complexity of lens design is for the most part hidden from 
users today. This is in part an effect of the ‘black-boxing’ that occurs as mechanical 
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devices become digital, passing beyond the skills base of the average tinkerer. Yet 
lens design, except in the use of exotic minerals in glass making, has not fundamen- 
tally changed in the transition from analogue to digital photography in the twenty- 
first century. While Eastman Kodak’s 1900 Box Brownie may be said to have started 
the process of black-boxing for consumer snapshots, professional and pro-am photog- 
raphers kept their interest in the technological properties of the medium at least as 
late as the 1950s, when advertising for anastigmatic lenses came to an end (despite 
the fact that all lenses had been anastigmatic for some decades by this point). The 
inference of the adverts was that photographers understood the optics of astigma- 
tism (the aberration which makes it impossible to have both horizontal and vertical 
lines align on the focal plane) and appreciated the engineering required—a mini- 
mum of three elements and two types of glass—to correct it. After the 1950s, lens 
design became even more a matter of branding, the domain of white-coated boffins, 
removed not only from the snap-shooter but for the most part from the professional 
photographer as well. The choices for those most intricately involved in the tech- 
nical aspects of the profession—cinematographers for example—were restricted to 
the choice of lens, or in extreme instances like that of Stan Brakhage, who rejected 
lenses altogether for Song XII (1965), choosing to film through the open aperture 
of a super 8 camera with no lens mounted (Wees 1992; Davenport 2001-02). In 
Brakhage and related practices we confront art premised on revolt against the uni- 
versality and invisibility of the grammar of the lens. We should consider the options 
for such a revolt today, as we migrate to digital projection. The engineering marvels 
we encounter should not distract us from increasingly relentless standardization, in 


direct continuity with the history of analogue projection. 


Digital Light Programming 


DLP (digital light programming), developed by Texas Instruments for digital projec- 
tion, is the most common form of projection today. DLP projectors use metal halide 
arc lamps, either in mercury vapour or in inert noble gas tubes, with infra-red reflec- 
tive surfaces to concentrate heat at the tips of the arc itself, where the intense light 
is produced, advancing on but sharing the same fundamental design as the old car- 
bon and xenon arc lamps. This lamp is designed to produce light at as many visible 
wavelengths as possible; that is, to give the closest technically available approxima- 
tion to pure white. The light is channelled towards the truly digital element of the 
projector, the DMD or Digital Micromirror Device. This is a chip (expensive models 
have three) containing up to 1.3 million digitally controlled mirrors (in 1280 x 1024 
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resolution machines), each of them 16 micrometers square, or about a fifth of the 
diameter of a human hair. The micromirrors, which fit on a DMD chip no bigger than 
the traditional photographic 35mm frame, are capable of ten degrees tilt, with 16 
millisecond response times, to reflect light towards or away from the projector lens. 
‘Black’ translates as a turn away from the lens; grey is produced by flickering at a 
higher rate than the basic 16 milliseconds as appropriate to the shade desired. The 
unused light has to be trapped, in a ‘light sink’, using the same kinds of light-absor- 
bent materials as camera housings, so that it does not bounce around the cham- 
ber, interfering with the information content of the video signal. It is only after this 
reflection that we can speak accurately of ‘optically programmed light’. 

Single chip DLP projectors use rotating filters to send three versions of an 
image—red, green and blue—to the screen, a process which clearly makes resolu- 
tion and registration extremely important if fringing and other artefacts are to be 
avoided. Filters also reduce the amount of light that actually gets to the screen. Thus 
the kinds of lens technologies we have mentioned are still vitally important to digital 
projection, and still control and shape the grammar of the projected image. High- 
end projectors use one DMD chip each for red, green and blue signals, using additive 
colour to make white, and absence to make black. Later models add a CMYK (cyan, 
magenta, yellow and black) ‘secondary colour’ wheel to provide a richer gamut of the 
subtractive as well as additive colours. These operate on the same basic principles 
as the tri-pack film used in cinema projection, where the light passes through suc- 
cessive layers of filters to garner the red, green and blue components of the image, 
corresponding roughly to the three types of cone cells in the human retina. Unlike 
cinema film, however, video signals (both analogue and digital) do not pass through 
a negative stage whose dyes are in the complementary subtractive colours, a pro- 
cess more or less universal in cinema after the 1930s, when experimenters with 
additive colour realised that the addition required far more light than subtractive 
colour. In all systems, however, colour requires more light than monochrome, which 
expresses only the ‘value’ or brightness of a scene. On the other hand, both types of 
projection find it hard to produce true blacks, given the scattering of light from adja- 
cent illuminated areas of the screen, as well as other light sources in the theatrical 
environment, such as exit and guide lights. Such matters are of paramount concern 
in gallery spaces not designed for projection, where the design and building of light- 
traps is as challenging an aspect of curation as is the management of sound overflow 
from one installation into the space of another. 

Apparent black—an optical impression of blackness as opposed to the actual 
absence of light—is easier to achieve when the image is denser. The proximity of 
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DMD cells to one another is far greater than the proximity of pixels in LCD (liq- 
uid crystal display) screens, giving a correspondingly greater apparent density. 
The additional brightness which, by contrast, makes blacks appear deeper, is also 
needed to make up for the difference in intensity between light sources (like the 
LED [light-emitting diode] backlights on LCD screens) and merely reflected light: 
reflection, no matter how finely silvered the screen may be, always absorbs some 
light, and reflects more away from the perceiver, so is always somewhat less effi- 
cient in maximising the light. Maximising the amount of light coming from the lamp 
itself is crucial. 

The same can be said of liquid crystal on silicon (LCOS) projectors, which split the 
white light from the halide arc prismatically, reflecting it from red, green and blue 
panels as desired, as in DLP projectors either sequentially (in single-chip projectors, 
relying on optical mixing in the viewer’s eye) or simultaneously (in the more expen- 
sive three-chip versions). LCOS pixels are formed from the same liquid crystals used 
in LCD screens, which untwist when charged to allow polarised light to pass through 
them, or absorb the light when uncharged, so appearing dark. In LCOS, crystals 
which respond to red green and blue signals are mounted on highly reflective pan- 
els. Older and cheaper LCD projectors do not reflect the light but beam it through 
LCD panels, in the process losing a great deal of the precious light generated by 
the metal-halide arc. In all three designs, LCD, DLP and LCOS, focusing the light of 
the arc lamp towards the projection lens is absolutely critical, and in all three the 
compound projection lens is tuned to produce a standard cinematic grammar. Even 
where the projection is at domestic scale, as in rear-projection TVs which provided 
the first market for LCOS chips, the competition with LCD and plasma screens for 
brightness and apparent contrast means that maximising the efficient use of light at 
all stages is a crucial concern. 

Much of the old work of the projectionist is automated in digital projectors. Gone 
are the hand-cut aperture plates of yore. It is impossible to get inside the lens assem- 
bly, and the trimming of the cinematic frame to remove the edge effects is done in 
DVD mastering and digital telecine. Levels of illumination and colour are controlled 
with a remote rather than by adjusting the lamp, housing and lenses; and a critical 
issue in projection, key-stoning (the trapezoidal result of distortion caused when the 
edge of image closest to the projector is shorter than that furthest away) is automati- 
cally reset by the projector without interference—in fact getting a digital projector 
to give anything but a four-square image can require some inventive interaction. 


The democratisation of projection, as is so often the case with visual technologies, 
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has come at the cost of a loss of control over the operations of the characteristic 


devices we use. 


From Searchlights to Laser 


The searchlight, the cine-projector’s sibling, makes evident a quality of projected 
light that would make it foundational for twenty-first century media. The earliest 
searchlights made their appearance at the Chicago World’s Columbian Exposition of 
1893, and appear to have derived some of their inspiration from lighthouses, that is 
as beacons to announce a presence, rather than as defensive devices for identifying 
attackers. A few even employed the very beautiful Fresnel lenses developed in the 
lighthouse world to concentrate the light, often of oil lamps, into parallel beams that 
would be visible in foul weather from ships some distance away. Searchlights also 
seem to have a heritage in focused lanterns used in navigating the often misty riv- 
erways of the Thames and Mississippi among other commercial waterways, as well 
as early attempts to militarise light in the Franco-Prussian War. The purpose of the 
condenser lens and the parabolic mirror shared by searchlights and cine-projectors 
is to produce parallel beams, with as little scatter as possible, making maximum 
use of the light from the lamp. In the cinema, this is at the service of the flicker- 
less, pin-registered clarity and brilliance of an image. In the searchlight, in its early 
use in commercial spectacle, the content was either the object iduminated—as in 
floodlighting today—or the weightless architecture of light brought into miraculous 
being over World’s Fairs and similar spectacular sites. This process meets its apo- 
gee in the intangible architecture of the Zeppelin Field devised by Albert Speer for 
the Nuremberg rallies, where the projected light attains the purity of light without 
image, light as sheer spatial organisation. In Speer’s searchlight architecture the 
technology is tripped back to its formal essence, light as the contentless medium 
which Marshall McLuhan proposed in Understanding Media—‘The electric light is 
pure information. It is a medium without a message’ (McLuhan 1964: 15). 

Lyotard, in his ‘Acinéma’ (1978), painted the apotheosis of cinema in the waste- 
ful, playful figure of a child playing with matches, a trope which echoes the solar 
excess of Bataille’s Accursed Share (1988). Its actual apotheosis is the light pillars 
of the Nuremberg rallies: the symmetry and order of classicism subordinated to the 
aesthetics of total command. In a chapter titled ‘Architectural Megalomania’, Speer 
offers his own description: 

The actual effect far surpassed anything I had imagined. The hundred and thirty 
sharply defined beams, placed around the field at intervals of forty feet, were visible 


51 


52 Sean Cubitt 


to a height of twenty to twenty-five thousand feet, after which they merged into a 
general glow. The feeling was of a vast room, with the beams serving as mighty pil- 
lars of infinitely high outer walls. Now and then a cloud moved through this wreath 
of lights, bringing an element of surrealistic surprise to the mirage. I imagine that 
this ‘cathedral of light’ was the first luminescent architecture of this type, and for 
me it remains not only my most beautiful architectural concept but, after its fashion, 
the only one which has survived the passage of time. (Speer 1970: 59) 

The conditions of Nazi Germany were certainly unusual, and it would be wrong 
to ascribe to the history of light a narrowly fascist ideology. Speer preferred a psy- 
chological account of Hitler’s personal investment: confronted by plans for immense 
projects like the Zeppelin Field, ‘Possibly at such moments he actually felt a certain 
awe; but it was directed toward himself and toward his own greatness, which he 
himself had willed and projected into eternity’, further noting that, ‘[s]uch tenden- 
cies, and the urge to demonstrate one’s strength on all occasions, are characteristic 
of quickly acquired wealth’ (Speer 1970: 69). The increasing devotion to the control 
of light should instead be read, as we may understand Nazism itself, as an expres- 
sion of an underlying demand for order which emerges both in the cult of efficiency 
(especially remarkable in the early twentieth century cult of the engineer) and in 
the rappel a l’ordre of populist politics in the 1920s and 1930s. Perhaps because the 
military technology of searchlights was tactically separated from the electrical grid, 
it was able, very early, to aspire to the autonomous development of the technology 
of illumination in its purest form, and provide the model for the commercial applica- 
tions which developed rapidly afterwards. 

Some decades later, lasers would take over as the medium of choice for dynamic 
urban architectures and new spectacular forms such as stadium rock. Light 
Amplification by Stimulated Emission of Radiation, laser, relies on controlling the 
random production of light typical of the sun, flames and incandescent lamps. Like 
flame and incandescent lights, lasers use the photons released by the movement of 
electrons between states of excitement. Unlike flame or incandescent bulbs, lasers 
use a form of amplification of light by passing a beam repeatedly between mirrors 
through a ‘gain medium’ (usually crystal, glass, metal or inert gases or semicon- 
ductors, but also liquids in so-called dye-lasers). In the gain medium the photons, 
released when electrons from atoms of the same composition and state of excite- 
ment release their energy, scatter as usual, but the paired mirrors encourage those 
photons travelling along their axis to continue bouncing to and fro, as they do so 
spurring other electrons to release their energy in the same wavelength as the one 


triggering the release. Thus the laser produces increasing numbers of photons of 
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identical wavelengths tuned to one another (‘in phase’). Once this reaches a critical 
quantity, photons travelling along the axis pass through the half-silvered mirror at 
one end in the form of an intensely focused beam whose wavelength is a function of 
the distance between the two ‘resonating’ mirrors. 

Different types of laser use a variety of materials, but it is worth recalling that 
the first optical lasers—the invention remains the subject of intense patent wars and 
claims for priority (Hecht 2005)—particularly the ruby laser, work in response to 
a source of incoherent light, which provides the initial excitation for the electrons. 
The typical source then, and in many contemporary designs, is a flash tube, electri- 
cal devices designed to produce extremely bright, extremely white light for a brief 
period. The pulsing light may have helped shape the staccato conception of laser as 
a digital medium. One characteristic links it back to the history of cine-projection: 
the fact that such light pumps are arc-lights, like the traditional cinema projector, 
a technology shared in single-use flash tubes (in the UK flash bulb) in photogra- 
phy. Between the single flash photography, the steady but mechanically interrupted 
light of a projector lamp, and the pulsing flash tubes of laser light pumps there is a 
direct continuity, one which however leads towards a more radical form of disconti- 
nuity than that evidenced by the interruption of intermittent motion and the shutter 
in the cinema. 

Lyotard’s figure of pure cinema as the child playing with matches was already 
inaccurate of cine-projection. In laser technology we can see, however, why Lyotard 
wanted to hang his hat on the theory of cinema as excess. As distinct from lasers, 
the controlled light of projectors is nonetheless a playful, human-scaled thing, like 
fire. In flame, free ions release photons as what even physicists call excess energy; 
something similar happens in the old arc lamps. In lasers, that excess is cancelled 
out by the controlled build-up of excitation and release, and the even more authori- 
tarian harmonisation of phase and wavelength. We might see Lyotard then as a 
late Romantic, continuing a tradition stretching back from Bataille to Nietzsche, 
Rimbaud, Blake and Hölderlin, in which agony and ecstasy pose themselves as escape 
routes from the advance of scientific rationalism and industrial standardisation. In 
critiquing the burning match model of cinema, we must be careful to avoid the sim- 
plistic opposition of excess and instrumental reason: spectacular waste is integral 
to consumer capitalism, even if it is in the carefully managed form of a carnival cir- 
cumscribed by discipline and anxiety, and formally orchestrated within arithmetic 
technologies like the laser. At the same time, we should recognise both that artists 
like Rafael Lozano-Hemmer have found extraordinary ways to release a democratic 


and liberatory potential in laser technologies, and that the child fascinated by flame 


53 


54 Sean Cubitt 


is only in danger because she is learning how to control the light. It is not control 
itself, but the normative power of universal standardisation around a meticulously 
arithmetic technology that requires our analysis. 

Light pumps produce very intense brief bursts of wide-spectrum incandescent 
light. The laser turns these into coherent waveforms, which differ from incandes- 
cent light not only because their colour can be controlled, but because their common 
phase means that they can function as a carrier wave in the manner of radio trans- 
missions, whose modulations are decipherable as signals. Intensely focused light 
could be used in many scientific instruments (for example to get an accurate mea- 
sure of the distance from the Earth to the Moon), and in such applications as laser 
surgery. Perhaps its most popular instantiation is in laser light shows, which now 
provide a common device for architectural displays, as found in Las Vegas and other 
urban entertainment and retail complexes. The proximity to Speer’s searchlights 
scarcely needs underlining. However, it is the signalling capacity of lasers that 
became the most significant aspect in transitions to digital media. Experiments with 
airborne transmissions proved too unpredictable for messaging: rain and fog dissi- 
pated the signal, even though they made the laser visible in air, the same dissipation 
as strategically employed in Speer’s searchlights. Laser pointers and range-finders 
as well as entertainment uses would continue the open-air technique in laser tech- 
nology. Meanwhile Cold War experimenters in the USA, UK, Russia and Japan con- 
verged on getting coherent light from lasers to travel through flexible glass pipes. 

Fibre optics work on the principle of total internal reflection, a phenomenon 
already observed and reproduced as a scientific amusement in 1844 (Hecht 1999: 
13-14). Here the light from an arc lamp is guided through the curving fall of a jet of 
water, the light waves bouncing from the inner walls of the jet, reflecting back from 
the meniscus. An 1881 patent suggested using the same phenomenon in glass rods 
to deliver light throughout an apartment building (Hecht 1999: 23). Coherent wave- 
forms were much better guided between surfaces when a glass rod of one refractive 
index was fused inside a glass tube of another, and the two heated and extruded to 
a fine filament. Already in use over short distances for gastroscopy in the 1950s, 
the nascent technology was wedded to lasers in the 1960s (Hecht 1999: 88-91). Ina 
return to cinematic technologies, confocal lenses—deliberate ‘imperfections’ func- 
tioning as series of lenses keeping light away from the walls of the fibre—guide and 
thus amplify the transmission of light through fibre: indeed Brian O’Brien, lead sci- 
entist at American Optical, who perfected a number of key fibre optic technologies, 
took a sabbatical from lasers to work on the Todd-AO widescreen cinema system. The 


proximity of cinema and fibre optic telecommunications reminds us that the control 
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of light is a constant goal of the emergent visual technologies of the twentieth and 
twenty-first centuries. In both, the varying qualities of different types of glass are 
crucial, as are the use of ‘doping’ to create specific optical qualities, the integration 
of gaseous states of matter between solids, and the critical development of pulsed 
lasers which matches the interrupted motion of cine-projection, giving both a com- 
mon control over the temporal as well as spatial dimensions of beams of light. 

The period of invention ended in the 1970s. Further developments improved the 
strength of signals over intercontinental distances, cheapened the manufacture of 
bundled cores of fibre, lessened the energy cost of interfacing, sending and receiv- 
ing equipment with the optical elements, and increased the speed and reliability of 
transmission in line with the massive infrastructural investment required to make 
fibre optics the core infrastructure of global digital systems in the twenty-first cen- 
tury. These technologies were in part a response to the threat of nuclear war in 
the 1950s and 1960s (optics are not damaged in the same way as electromagnetic 
equipment in the magnetic pulse of an atomic explosion), but were also the heirs of 
the extraordinary flowering of glass technologies in the nineteenth and twentieth 
centuries, a field which at first glance seems quite discrete from the development of 
mechanical and later electronic imaging, but which on reflection is clearly integral 
to both. The common theme is control, the production and distribution of what laser 
engineers refer to as coherent light. 

To a certain extent, the title of the present work, Digital Light, is a misnomer, in 
that light is light, and only becomes digital in the process of a systematisation which 
drives through the history of projection and fibre optics. The base principle here is 
less electronic than formal: the use of enumerable units to parcel out the otherwise 
formless omnidirectionality of light. The irony is that in one remarkable account of 
light given by the eleventh century divine Robert Grosseteste, light gave form to the 
created universe. Taking the Hebrew of Genesis literally, Grosseteste argued that 
the phrase ‘without form and void’ implied that the world prior to Creation was a 
formless mass of stuff. God’s first words, ‘Let there be light’, divided the formless: 
night and day, land and ocean, spreading form by distinguishing, in the double sense 
of dividing and making visible (Crombie 1953). But the divine radiance is of another 
quality to the ordering brought about by digitisation. The wave-particle argument 
that began at the time of Huygens and Newton (Dijksterhuis 2004) helped engender 
a new interest in glass as a scientific instrument for the analysis of light, and fore- 
shadowed the partition which would overcome it at the end of the nineteenth cen- 
tury. At this stage, the quantum physicists, despite their allegiance to the dual form 
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of wave and particle, opened the doors to technologies which have depended on the 
unit of light as a measurable, accountable and manageable quantity. 

Division, initially undertaken in the shutter mechanism of cine-projection, is 
integrated into electronics in rough form in the cathode ray tube, increasingly con- 
trolled in liquid-crystal displays, and today controlled with extreme efficiency in 
a fibre optic infrastructure whose norms are governed by the standards promul- 
gated by the International Electrotechnical Commission (IEC), the International 
Organisation for Standardisation (ISO) and the International Telecommunication 
Union (ITU). These bodies, mostly originating as professional associations of engi- 
neers charged with ensuring the interoperability of national systems, organised 
their membership on national lines, especially when ITU and ISO entered the United 
Nations system. More recently, they have been increasingly influenced by corpora- 
tions (for example, see MacLean 2003), including manufacturers, distributors and 
retailers, as well as by the competition regulation of the World Trade Organisation. 
Virulent patent wars have broken out over visual technologies in the wireless world 
(Reuters 2011), suggesting that under the guise of competition, there has been a 
powerful convergence of technical specifications in the field. Predominant in the 
global long-distance market and increasingly in fibre-to-the-door initiatives, the 
fibre optics backbone of telecommunications networks, has been successfully pre- 
served from such disruptions by the need to ensure a globally functional network 
from which all carriers, suppliers and equipment manufacturers benefit. Under the 
auspices of both standards bodies and the Fiber Optics Association, which provides 
certification and training worldwide, increasing convergence has moved the indus- 
try away from the invention phase of the 1970s towards a far more restrained and 
constrained phase of relatively minor innovation, as the technology has become the 
essential structural medium of planetary communications. In this, fibre optics regu- 
lation follows the lead of the Academy of Motion Picture Arts and Sciences, which, 
in addition to the Oscar ceremonies, is responsible for standardisation in the film 
industry since the late 1920s, and the adoption of the Academy Standard screen 
aspect ratio. Standardising the number of sprockets per frame, the motor speed of 
projectors, the dimensions of the vision and soundtrack areas of the filmstrip, and of 
course aspect ratios made the global export of films a major industry in the wake of 
World War I (Thompson 1985). The comparison between the wireless and fibre optic 
industries demonstrates that, while there is still competitive advantage to be had 
in the as yet to mature mobile market, in the infrastructural domain, whether tele- 
communications or cinema, standardisation is a goal rapidly sought out and care- 


fully guarded. 
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The specificity of the unit measure is less obviously important, save as a logical 
extension of digital communications, which depend on the distinction between open 
and closed channels, whether the flow being controlled is of electrons or photons. 
The distinction, read in machine code as that between ones and zeros, is however 
less clear in engineering terms, where the residual refracted light of a previous ‘one’ 
will always interfere with the present ‘zero’: absolute dark, like the absolute absence 
of electrical charge, is a physical possibility only under extreme laboratory condi- 
tions. To produce the effect of difference, which is the crucial feature of digital infor- 
mation, requires careful manipulation of the material form of the wave constituting 
the passage of a photon through the fibre. The waveforms carrying digital signals 
through fibre optics share their shape with more and more universally standard- 
ising transmission protocols within as well as between devices. As information is 
promulgated between components of a digital system—from the photosensitive chip 
of a camera to its storage, for example, or from that store to the editing applica- 
tions that will work on it, and again from there to encoding for transmission or disc- 
recording—whether as light or voltage, data is encoded in waves which are tailored 
for the on-off intelligence of the digital system in the form of the ‘square wave’. 

Viewed at the microscopic scales at which they occur in fibre, it is clear that 
square waves are not abrupt switches between on and off but exhibit steep ramps 
between the on and off states. This quality is eliminated from the reading end of 
the process, which consistently sees only one of two states. To ensure this happens, 
the communication must be damped to eliminate ‘ringing’ effects, in which the 
remnants of the natural sine wave reverberate with each other and the materials 
through which they pass. As so often in digital media, the process involves auto- 
mated approximation algorithms. In this case the Fourier transform used to approxi- 
mate the two signal strengths to zero and one is itself responsible for the discontinu- 
ities which produce the ‘ringing’ effect, which in turn must be damped to maintain 
the illusion of a unitary sequence. 

If the unit-counting of digital media matches the commodity economy, which 
maintains the discreteness of each item traded, the approximations match the mana- 
gerialist or bio-political mode of contemporary power. As we all know, transmission 
does not differentiate between different content types, only between units (packets, 
pixels); neither in packet switching nor in routers. All that is important to the TCP/ 
IP (transmission control protocol / internet protocol) suite, and to the functioning of 
any digital device, is that data be ordered in units, and that the units can be handled 


as if coherent. Processing—such as the Fourier transform or the square wave—only 
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completes the universalisation of statistical averaging and unit enumeration as the 
core formal qualities of the information age. 

The principle of control emerged in the design of projectors, where control over 
the random flight of photons in every direction and at every wavelength from any 
source had to be tamed if a tiny frame of film-stock was to be projected onto a huge 
screen using any realistically achievable quantity of energy. Shaping the beam 
through parabolic mirrors and condenser lenses structured other informational 
uses of light in searchlights and lighthouses. The irony is that the light which is so 
powerfully constrained in these systems, and which is even more tightly corseted in 
the internal reflections of fibre optic waveguides, is the medium of twenty-first cen- 
tury telecommunications and network media. The organization of light in projection 
has found in the twenty-first century an even more precise conformation of light to 
the requisites of an instinct towards order that has become totalitarian. 

Late in his life, affected by the cases of shell shock he had witnessed after 
World War I and perhaps even more so by the rise of Nazism, Freud proposed the 
idea of the death instinct. From 1920 to 1930, in Beyond the Pleasure Principle to 
Civilisation and its Discontents (Freud 1961a; 1961b), he developed a dense theo- 
risation of a drive towards entropy informing both personal and social structures. 
The child’s fascination with the flaring match illustrates that this drive, like every 
other, easily oscillates between positive and negative poles: between the drive to 
destruction and the drive towards order. If at one extreme, this leads to the repeti- 
tion compulsion and to a collapse into the inorganic, at the other it leads to the kind 
of pathological command over physical reality which, paradoxically, no longer frees 
us from the contingency of the laws of physics but enslaves us to their organisation 
in global technical systems, themselves expressions as well as formative vehicles of 
an increasingly global order. The historical realisation of Kantian freedom from the 
laws of nature as foundation of the ‘cosmopolitan intent’ has in actuality come about 
not through international law, as he imagined (Kant 1983), but through the kind of 
normative technical structures underpinning the pursuit of coherent light. This pur- 
suit was undoubtedly experienced in the beginning as an autonomous movement of 
the nascent techno-science of the late nineteenth century, but has become rigor- 
ously integrated into the hardwiring of contemporary global infrastructures. It will 
be one of the key challenges of the twenty-first century to develop both radical ways 
of working within this already ossified system, and to invent new modes of working 
with light that involve not simply freeing it, as an entropic gesture, but finding new 
ways to create in partnership with light, rather than through its enslavement. 
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CHAPTER 3 


HD Aesthetics and Digital Cinematography 


Terry Flaxton 


This chapter is accompanied by a series of online interviews entitled 
‘A Verbatim History of the Aesthetics, Technology and Techniques of 
Digital Cinematography’. This online resource seeks to circumscribe 
and circumlocute the wide variety of interests and usages of incoming 
digital media with specific relation to the effects of increased resolution 
being offered by emerging digital technologies and can be found here: 


www.visualfields.co.uk/indexHDresource.htm 


In April 2007 at the National Association of Broadcasters convention in Las Vegas, 
High Definition Video changed forever. Whereas previous HD cameras had cost half 
a million dollars, Jim Jannard, a sunglasses manufacturer from Canada, had man- 
aged to develop a new camera called the ‘Red One,’ retailing at $17,500. This devel- 
opment signalled a change in the production of High Definition as it had been first 
conceived and named. The original title—‘High Definition’—was meant to signal a 
change from standard resolution digital video and align the new technology with 
film, giving it more of a sense of quest than analogue or digital video, more of a 
sense of flight, a sense of the arcane and the hidden, thus producing something to 
aspire to and engendering a sense of being elite—very important for the Directors 
of Photography, those captains of the ship heading towards the image horizon and in 


turn, evoking some of film’s prior sense of mystery. Now we are in the stable years of 
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‘HD’, the title ‘HD’ has become misleading, mainly because it refers to a line struc- 
ture that was pertinent to the analogue age (and related to television) but which no 


longer appropriately characterises the current aspirations for the medium. 


A High Definition Image to Recall 


I want to introduce an image that may be useful when thinking of HD: as the light 
falls at dusk and you are driving along, you might notice that the tail lights of the 
car in front of you seem much brighter than in daylight, and the traffic lights seem 
too bright and too colourful. The simple explanation for this phenomenon is that your 
brain is switching between two technologies in your eyes. The first technology is 
the rods (inherited from our distant ancestors), which evolved for the insect eye to 
detect movement, and are numerous at around 120 million. Through them you see 
mainly in black and white. The second technology is much more sensitive to colour: 
these are the cones, which are far less numerous at around 7 million. 

Colour is a phenomenon of mind and eye—what we now perceive as colour, is 
shape and form rendered as experience. Visible light is electromagnetic radiation 
with wavelengths between 400 and 700 nanometres. It is remarkable that so many 
distinct causes of colour should apply to a small band of electromagnetic radiation to 
which the eye is sensitive, a band less than one ‘octave’ wide in an electromagnetic 
spectrum of more than 80 octaves. 

Human trichromatic colour vision is a recent evolutionary novelty that first 
evolved in the common ancestor of the Old World primates. Placental mammals lost 
both the short and mid-wavelength cones. Human red-green colour blindness occurs 
because, despite our evolution, the two copies of the red and green opsin genes 
remain in close proximity on the X chromosome. We have a weak link in our chain 
with regards to colour. We are not 4 cone tetrochromats; we have three and in some 
cases only two—in extremely rare cases we have one. 

So, there are two technologies—rods and cones—between which there is a physi- 
ological yet aesthetic borderland. Keeping this idea in mind, if we apply the poten- 
tial misreading of eye and mind not to colour, but to our ability to recognize differ- 
ent resolutions, then a similar potential sensorial confusion is possible in the higher 
resolutions of which we are now capable. In my own experiments with capture and 
display, it is becoming apparent that a viewer experiences a sensation similar to 
the illusion that there is more colour at dusk when a certain level of resolution is 
reached. At that borderline between the lower resolution and the higher resolution, 


a fluttering occurs as we engage in this step-change of resolution. I have found that 
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at the lower level there is less engagement, as measured by the duration the audi- 
ence is willing to linger with an image, and at the higher resolution there is more 
engagement. This is evidence of a switching between two states in the suspension 
of our disbelief—with higher resolutions eliciting more visual fascination. What is 


really interesting to me, as an artist, is the boundary between the two states. 


The Figures 


After the invention of television, it took many years to be able to record the ana- 
logue video image. This was finally accomplished through creating a scanned rasta 
of lines and inscribing what information was present in each line. This was the strat- 
egy of analogue video in its two main forms: PAL and NTSC. When computers began 
to take over, scanning became obsolete (having only been necessitated by the limi- 
tations of magnetic control of electron beams and glass technology at that time); 
so a form of inscribing and recording the information that was independent of the 
scanned rasta but was grid-like—digital in form and mode—took over. This became 
the now familiar grid of pixels that every camera sensor has. A progressive image 
sensor is like a frame of film in that it is exposed in one go, unlike a scanned image, 
which takes time. But there are many issues with the technology that make it unlike 
film (like needing to empty a CCD of charge, line by line, or a CMOS chip in one go). 
Each chip is constructed of many individual photosites that are single light sensi- 
tive areas. These then produce information in the form of packets of data that are 
in turn represented on screen by a changing luminosity and colour identity via a 
pixel of display. 

But let us step back to the moment when analogue video began to give way to the 
first digital forms. Digital Video was then transforming analogue technology and 
this moved us closer to High Definition technology. It had 720 x 576 pixels to emu- 
late the 625-line system in analogue video (in PAL at least). It required anamorphis- 
ing to enable a 16:9 ratio from a near 4:3 pixel count. It confused many of its early 
adopters because it seemed to disintegrate through multiple copies—yet of course 
Digital Video was heralded as non-degenerative. The fact was that it was only par- 
tially digital, but it gave us a run for its money. 

The earliest forms of High Definition were analogue at around 1250 lines, but 
being on the cusp of the digital revolution, HD soon abandoned the early analogue 
HD forms as it became digital. In economic terms this meant that the early European 


systems were being financially trounced by the Japanese and American systems, so 
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the standard eventually became 1920 x 1080 pixels (which had a relationship to the 
analogue NTSC format). 

Standard HD is known as having 2k resolution because it has a resolution of 1920 
x 1080 pixels (1920 is near 2000). This has a 16:9 or 1.77:1 aspect ratio, which is com- 
mon to LCD, LED and plasma television design. Cinema style HD has been called 
Electronic Cinematography—it is also 2k but has 2048 x 1080 pixels (or sometimes 
2048 x 1024). This has a 2:1 aspect ratio. The academy version of 2k has 2048 x 1536 
pixels, which is 4:3 aspect ratio. So there are varying requirements concerning the 
number of pixels in an electronic cinematographic image—agreements still have to 
be made as to exactly what numbers are involved though this is getting closer with 
the Academy Colour Encoding System (although this is primarily a system to deter- 
mine standardised colour through different media, it will have a knock on effect 
with regard to resolution). There is also one other important difference between 
Standard or HD resolutions (which are governed by proprietary formats) and 
Electronic Cinematography. Proprietary forms of HD are generally processed (or 
data reduced and processed) in camera, whilst Electronic Cinematographic forms 
are processed mainly in the post-production house. 4k is 4096 x 2160 pixels (2:1) or 
4096 x 2304 (16:9), and 8k is 7680 x 4320 (16:9)—this last is NHK’s Super Hi-Vision. 
In conversations with leading designers in the subject area I have established that 
far higher resolutions than 8k are in development. 

It is possible to record most of these formats in a compressed version on a form 
of memory card, but currently, the highest level of 2k HD image capture requires 
recording onto solid state discs—and not just any solid disc, but a Redundant Array 
of Independent Discs—a RAID (the exception is Sony’s SR deck, which records data 
on tape). If you want to record 1920 x 1080 pixels uncompressed, then you need read 
and write speeds of over 440 Megabytes (Mb) per second. The average old style 
hard drive reads and writes at around 35 Mb —hence you need quite a few of these 
(though solid state drives record much higher rates you still need several of these 
too). To understand the idea of the RAID, imagine the following: If I throw you a ball, 
you might be able to catch it. If I manage to throw you twenty balls at the same time, 
you have no chance. If I throw twenty balls to you and another nineteen friends—you 
have a chance of catching them. A RAID Array uses a group of discs to catch large 


amounts of data. 
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Criteria for Digital Cinematography 


Until recently many manufacturers used small chips of around half to two thirds 
of an inch in size—each camera used three chips for three colours to reconstruct a 
colour image (red, green, blue). The chip size was a hangover from 16mm film that 
generated large depth of field from the smaller optical pathway. When Red Cameras 
introduced the Red One it began with a 35mm-sized single sensor that used a Bayer 
filter to extract colour information. Bryce Bayer invented this system for Eastman 
Kodak in 1976—primarily to extract information from scanned film images. 

With the above in mind here is what I consider to be the governing principles for 


Digital or Data Cinematography: 


a) The optical pathway is 35mm or above (derived from technical and 
industrial limitations possible at the time of origination for manufac- 


turing photo-chemical negative). 


b) It generates a progressively based image flow relating to a specific 
time-base as opposed to an interlaced image flow (one full frame of 


information at a time rather than a field-based workflow). 


c) Like one of its predecessors, film, it holds the image in a latent state 
until an act of development (or rendering) is applied—but unlike film it 


is non-destructive of its prior material state. 


d) Its capture mechanism, though generating a non-destructive, non- 
compressed data pathway from which an image can be reconstructed, 


does not have this as its sole intent as a medium or method of capture. 


These latter three qualities are also base characteristics of many other developing 
digital technologies—for instance real-time mapping of environments requires a 
capture of infra-red imaging sources (cameras used as sonar devices) running at 
around or above 25 fps. Using this criteria, Digital Cinematography is about more 
than just capturing images—it’s a portal onto the digital landscape so far unexplored 


due to its apparent function as an image capture medium. 


What Was the Future, Is Now the Present 
The future is governed by Gordon Moore’s Law, formulated in 1965, which proposes: 


The complexity for minimum component costs has increased at a rate 
of roughly a factor of two per year ... Certainly over the short term this 


rate can be expected to continue, if not to increase. Over the longer 
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term, the rate of increase is a bit more uncertain, although there is 
no reason to believe it will not remain nearly constant for at least 10 
years. That means by 1975, the number of components per integrated 
circuit for minimum cost will be 65,000. I believe that such a large cir- 


cuit can be built on a single wafer. (Moore 1965: 114-17) 


In 1975, Moore altered his projection to a doubling every two years (Moore 
1975: 11-13). 

In the early 1990s I first came across analogue HD systems (including the 
European MAC system which used 1250 lines). In 1999 I shot a short on the Sony 
900, which was post-produced by Du Art in New York and then output to 35mm for 
display at festivals. In 2003, NHK, or the Japan Broadcasting Corporation, conducted 
an experiment that linked a prototype 8k 16:9 aspect ratio camera to 18 one-hour 
data recorders. The subject of the test was a car ride lasting three minutes. In order 
to capture it, the SR data recorders ran so fast that they went through one hour’s 
worth of recording during the three-minute shoot—all 18 of them. The resolution of 
the projected image was immense: imagine a computer display set at around 1280 x 
1024 pixels—then imagine a screen of some twenty-seven feet long with the resolu- 
tion proportionately more detailed. A little more than 100 years earlier, this tech- 
nological moment had echoes of the Lumiére Brothers’ screening in January 1896 
of a train arriving in a station. At the NHK screening, the Japanese audience were 
reported to have found the experience so overpowering that many of them experi- 
enced nausea. Both of these tales are at the same time apocryphal, yet pleasing. 

So now we can place a computer image on a screen of cinema proportions with 
equivalent or more than cinema resolution (at the time of correction Alfonso Cuaron’s 
Gravity has opened to wide acclaim - a feature of over 90 per cent computer graphic 
content). So, imagine what it would look like—in fact what it would feel like, if the 
very high density of pixels shot in the NHK experiment were then displayed across 
that screen as cinema—in the NHK experiment the possibilities of deep engagement 
and belief in the experience seem to have lead to a physiological reaction. Since this 
experiment, Super-Hi Vision has been streamed live between Tokyo and Osaka—but 


of course that act required a high amount of compression. 


Technology and Compression 


The high levels of data produced in creating digital cinematographic images should 
beg a primary question in the reader’s mind with regard to how we actually achieve 


data capture. Any serious understanding of High Definition / High Resolution 
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technologies requires a basic understanding of ‘compression’ (and this is without a 
deeper discussion of ever increasing colour bit depth where veracity of colour and 
a larger data collection is required to pursue the twentieth century’s primary aes- 
thetic project, that of realist and hyper realist intent). This question should be: How 
do we accomplish so much with so little? 

Light is focused through a lens onto a charged coupled device or sensor, which 
then emits electrical impulses that are reconstructed as data. Very early on in 
video production, a question arose for designers when far more data was generated 
through this process than was recordable. It was from this problem that the idea of 
throwing ‘unnecessary’ data away took hold. A strategy that commercial producers 
utilize is that of adopting the idea of GoP structures to compress images—and this 
practice underpins not only low-level HD recording in camera, but transmission of 
images over the Internet. 

GoP is short for Group of Pictures. The first and last frame in a group of pictures 
contain all the information: each succeeding picture only contains the changes in 
the information. If a person is photographed against a background, there is no need 
to resend the background information again and again—only the information about 
head, mouth and eye movements. You can see the affects of GoP structure effects 
when you watch the blocky artefacts in DVD or Blu-ray, or HD transmission occur- 
ring—there is a regular beating in the change of the blocks. HDV, P2 and AVC cam- 
eras use this system to record images and it is often criticized for being unable to 
handle motion well. Clearly the shorter the GoP structure, the better this system 
will handle motion. 

The traditional photographic camera manufacturing companies have recently 
got on the bandwagon, taking advantage of the convergence of still and motion pic- 
ture imaging. Some DSLRs, but not all, had the benefit of a 35 sized single sensor— 
but with limited writing speeds were restricted to GoP structure compression. By 
the end of 2011 Canon brought out the Canon C300 to fully enter the motion imag- 
ing market place. However, in an act that could be seen as an attempted spoiler, 
Red introduced the Scarlet X on the same day at half the price and like its anteced- 
ent, the Red One, with full frame recording. Some fans of Red complained that the 
Scarlet was simply a diminished version of Red’s Epic camera, one that did not value 
the rapid developments in the area. Others appreciated the gesture of the possibility 
of the mass availability of the technology. 

Other motion image manufacturers lead by Red, Arri, Panavision and now 
joined by Sony and Panasonic, realize—mostly through pressure by the cinemato- 


graphic community—that one of the baseline rules necessary to achieve True Digital 
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Cinematography is to keep all of the data in order to properly describe what is in 
front of the lens. At the time of writing there are no longer any manufacturers of 
photo-chemical based film equipment. 


So how have impossible amounts of generated data now become recordable? 


Wavelets 


An accumulation of data in a camera is a representation of the original scene and 
all representations have levels of veracity with regard the original. Previous SD 
and HD cameras use a software technology based on Jean Baptiste Joseph Fourier’s 
Discrete Cosine Transforms (DCTs), which break up the image data into tiles, so that 
each can be treated independently. Recently though, we have seen the arrival of 
Fourier’s Wavelet Transforms. The theories involved were in place by 1807 but not 
truly understood until about twenty-five years ago. Wavelets have helped prise open 
a Pandora’s box. Here is a description by the astrophysicist Amara Graps: 


Wavelets are mathematical functions that cut up data into different fre- 
quency components, and then study each component with a resolution 
matched to its scale. They have advantages over traditional Fourier 
methods in analyzing physical situations where the signal contains dis- 
continuities and sharp spikes. Wavelets were developed independently 
in the fields of mathematics, quantum physics, electrical engineering, 
and seismic geology. Interchanges between these fields during the last 
ten years have led to many new wavelet applications such as image 
compression, turbulence, human vision, radar, and earthquake pre- 
diction. (1995) 


Discrete Cosine Transforms are a sort of ‘one-size-fits-all’ response to data—a 
thuggish response requiring intensive computation. This is in contrast to Wavelet 
Transforms, which interrogate the data coming through them and find the best 
response from within their algorithm. In effect they intelligently address the data 
to get the best out of it, while using less computational power. As one Director 
of Photography put it on the Cinematographers Mailing List: ‘ummm, wavelets 
good, DCT bad.’ 

Contemporary cameras and post-production systems have been designed with 
DCTs in mind, and the manufacture of the relevant devices, cameras, proprietary 
editing and storage systems has been designed and marketed to recoup the large 
amounts of costly research that has been expended by big corporations. It has sim- 
ply not been in the interests of the bigger corporations to switch over to the new, 
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more elegant technologies. Yet the pressure exerted by the maverick Red Camera 
Company had telling effects on corporations like Sony, who marketed the F23 and 
the F35, followed by Arriflex with their Alexa camera. Sony’s naming system bears 
some investigation: The F23 used a two-thirds inch chip, the F35 a 35mm sized sen- 
sor and in 2011 Sony launched the F65, a possible reference to 65mm, which uses an 
8k chip that generates a truer 4k image. To obtain a true register at such resolutions 
one must take two samples to derive a value—this is know as the Niquist Shannon 
Sampling Theorem—an important factor in Modular Transfer Function or MTF. 

With a nod towards Ivan Illych and his studies around systems disorders— 
in his case with regard to how much allopathic medicine cures and how much ill- 
ness it induces (known as iatrogenesis), we must also understand that any sys- 
tem has a limiting element within it. In electronics this is referred to as Modular 
Transform Function. 

Modular Transfer Function describes a chain of delivery from capture to display 
where resolution is defined by the lowest resolution link in the chain (like plumbing 
where flow is derived from the thinnest of the pipes in the system). For instance in 
Apple’s Final Cut Pro Studio 3, the base architecture limits resolution to 2000 lines. 
If you shoot 4k, edit through Final Cut and then display on a 4k monitor—the MTF of 
the system is 2k (however, new iterations of that manufacturer’s software and other 
manufacturers of post software are using resolutions up to 4k and above). With the 
early Red One’s 4k sensor, because of various issues with the Bayer Pattern filter, 
the MTF of the camera is around 2.8k/3.2k—but of course if you use a lens with less 
than 2.8k of resolution, then whatever the lens resolution is would then be the limit 
of the system. 

Having said all of the above, a Wavelet Codec, because it deals with curves 
rather than angles, can intelligently reconstruct large amounts of data from very 
small samples. Red’s own proprietary Redcode 28 can construct a kind of 4k image 
using only 28 Mb’s of recording speed. (RC 36 and RC 42 use 36 Mbs and 42Mbs 
respectively.) A lossless 2k image should record at 1 gigabyte of data per second 
(dependent on bit depth etc) so the very idea of recording 4k in 28mbs is astounding 
and this demonstrates the efficacy of Wavelet Transforms. 


The Contemporary Argument 
About HD Image Control 


Currently, film DPs are still brought in to light 2k productions. Though they 
are becoming familiar with the thinking in Digital Cinematography, previously 
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electronically trained people were brought in to hold their hands; the common 
ground between the two is the attitude that ‘preserving data is all’. At one meeting 
of the British Society of Cinematographers, there was much wailing and gnashing of 
teeth as film-oriented DPs stressed their concern over the lack of dependability of 
the production chains that eventuate in an image. It was argued that it is currently 
possible to send your data through the same equipment at two different facilities 
in the same city and obtain different colorations of that image. It has taken 100 
years within the practice of photo-chemical film to obtain dependability in the chain 
of production, and of course the ability of a cinematographer to get that little bit 
extra, that un-definable advantage in their image making is what adds value to their 
reputation. However, at the moment, the terrain of Digital or Data Cinematography- 
based production is still feared because that level of control has yet to be fully real- 
ized. Having said that, the American Society of Cinematographers have instituted 
their Academy Color Encoding Specification and Image Interchange Framework, 
the intent of which is to introduce full calibration of the process—bringing images 
from both data and photo-chemical capture, through a reliable framework, right 
through to display. 

Within contemporary cinematographic aesthetics, whether in film, analogue or 
digital video, or Digital Cinematography, there are a series of tactics used to ‘say 
something’ with light. If listed, these tactics become mundane: a warm look for 
safety and comfort, blue for night, uncomfortableness, then into the greens, sodium 
or magentas for inducing the uncanny, threat and alienation—and so on. There are 
some DPs like Vittorio Storaro, who shot Apocalypse Now (Francis Ford Coppola, 
USA, 1979), who step outside of these prosaic colour values. Storaro has his own 
narrative colour system which works for him and like an extreme system of bidding 
in the game of Bridge, not too many people can operate it, nor many understand it. 
Yet many DPs agree, Storaro has ‘the touch’. Whereas Storaro works with colour and 
light, and the physiology of light enmeshed with a psychological narrative, Conrad 
Hall (born in 1926 and passed away in 2003: American Beauty, Sam Mendes, USA, 
1999; and The Day of the Locust, John Schlesinger, USA, 1975) thought differently to 
Storaro. His inventiveness and commitment was to the photographic within the cin- 
ematic arts. As Hall traversed the boundaries of contemporary wisdom about what 
constitutes good exposure, he influenced a whole generation of DPs. Hall knew that 
the still image captures something that cinematography rarely does; he was there- 
fore concerned with finding ‘the photographic moment’ amidst the flow of images. 
He tried to find the extraordinary within the ordinary. In this quest, Hall pushed 


film exposure to the limit; this kind of treatment would be ruinous to HD because 
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it does not yet enjoy the exposure of highlights that film does. However, with the 
arrival of Digital Cinematography this now becomes possible in some ways, though 
with the coming arrival of Higher Dynamic Range Capture and Display we shall 
soon be able to capture and display close to the dynamic range of the human eye and 
brain system. 

When I say ‘close’, within the human system we are capable of seeing fourteen 
orders of magnitude from the brightest day to the darkest night, but available to us 
at any given time are five orders of magnitude which we slide up and down the scale 
of sensory input. 

We are all familiar with how our eyes adjust to bright light, or conversely to the 
dark—that’s us making our five orders of sensitivity respond to the spectrum of light 
which occupies fourteen orders of magnitude in total. HDR Capture shoots two or 
more simultaneous images, of both highlights and shadows, then merges the result 
and takes the best of both, in order to display a higher dynamic range image. When 
displayed on contemporary displays, whether LCD, Plasma, or CRT, the dynamic 
range of the display space is only between two and three orders of magnitude; hence 
the description of the image as looking ‘plastic’ by many DPs. However, the HDR 
image should be displayed on an HDR display, which is capable of the same level of 
instantaneous orders of magnitude as the eye/brain. The common response to these 
new screens is that (to paraphrase) ‘it is like looking through a window, with all the 


attendant clues related to depth’. 


Looking Back in Wonder 


On a Ridley Scott set in 1983, as he shot the famous 1984 Apple Mac commercial, I 
was shooting the ‘making of’ material for Apple (on NTSC betacam video—on tubes). 
At that time it was not possible to see what kind of images you were obtaining via 
the medium of film. For that reason the cinematographer, through experience, would 
be one of the only persons on the set who knew roughly what they would be getting 
back in the next day’s rushes. As we were viewing back our video rushes on our pro- 
duction monitor, checking focus and exposure, I became aware that about 20 people 
were standing behind us, quietly looking over our shoulders. Usually the film rushes 
would come back the next day to be viewed by the more select in the hierarchy. The 
two groups stared at each other—two alien tribes at war—film and video. But this 
was a film crew that had never before seen what it had been shooting at the same 
time as shooting it. One of them grinned in pleasure at seeing our footage and sud- 


denly, like the German and British troops in the World War I downing their rifles on 
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Christmas Day and playing football together, we were friends. From then on they 
stopped being hostile to us, even sometimes offering to move lights to let us have 
some illumination—bearing in mind that lights are sacrosanct in film. 

Historically, however, in the clash between film and video, the professional film 
users were seen as artists and craftsmen and video users were seen as artless—even 
though video art was superseding experimental film at that time. Video was obtain- 


able yet without atmosphere; film was arcane, it was a quest in itself, it had kudos. 


Achieving the Filmic Look 


In film production, because film stock and lenses eventually had become so sharp, in 
order to impose an atmosphere, cinematographers have had to constantly distort the 
colour standards and definitions of film stock. ‘Atmosphere’, like popcorn, shares a 
quality that allows the easy suspension of disbelief. If film manufacturers say that 
development should occur at such and such a temperature, then heating up or cool- 
ing down the developer by a few degrees is a means by which the colour, grain or 
exposure may be changed. 

Here is the rub for Digital Cinematography: to get a look from a clinically clean 
medium you have to distress the image and therefore lose data, and as we’ve estab- 
lished, DPs really don’t want to distress an image that is already distressed by being 
compressed. If you do work on the image in camera, as the traditional film DP has 
tended to, then you limit how much data is recorded—you have to work in the colour 
matrix. If you crush the blacks to get a look you automatically reduce the data that 
is output into the image display. So current practice is to do very little in camera, 
so that every bit of data is carried back into post-production, where the work on the 
image—the grading—can begin. But I contend that when you really look at images 
produced like this, you’l] see a thin patina over the image and the ‘look’ itself is not 
inherent within the image. I’ve spent thirty years shooting video, as well as film, 
and I know it’s possible to generate the look within the image. It is my contention 
that simply to light well and to leave everything to post-production is an abrogation 
of the DPs responsibility as a creative artist. I have many friends in the cinemato- 
graphic community that now believe this to be true. 

Original electronic imaging was analogue in form—as was film—yet the formula- 
tion of the capturing of an image was different from film. Film has wide latitude— 
one can make an intelligent ‘mistake’ and rework the material and formulate a sense 
of ‘atmosphere’ within the image. This is commonly known as ‘the look’. Analogue 


video was clean and clinical, and you had to get the exposure right—in the early 
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days, if you didn’t get exposure correct, then you didn’t get focus. Colour itself was 
grafted onto an already set formulation of image capture. I shot one of the first fea- 
tures generated on video and transferred to film for theatrical distribution; this was 
Birmingham Film and Video Workshop’s production Out of Order (Jonnie Turpie, 
Birmingham Film and Video Workshop/Channel 4/BFI, 1987), and I approached the 
task by imagining video as being like a reversal stock—with very little latitude for 
mistakes in exposure. Nevertheless, what had I to lose? I tried various colour experi- 
ments—for instance, at that time creating a white balance that was off-white was not 
generally done. I discovered that not only could you tip the white balance towards 
the corrective areas of colour (blue and orange, cold and warm), you could also tip 
the balance of colour into any complimentary area—for instance, corrupting the look 
towards the purple to induce green and any variation to be found around the colour 
wheel. The transfer to film at that time seemed adequate, but when compared to 
today’s digital transfer techniques, it was not good in terms of colour. 

With the advent of Digital Cinematography something very important has hap- 
pened with image capture. In both photochemical and digital cinematography, until 
the image is developed, the image resides in latent form in both the silver halides 
and the un-rendered data. Development—the bringing forth of an image in film—is 
similar to the rendering of an image in the electronic domain except for the fact that 
the materialised film image is negative and in digital cinematography the material- 
ised image is positive. The nature of that difference requires fuller investigation at 
another time. 

For now, it is important to note that, in the latter, colour is within the bit-depth 
of electronic data and is therefore an integral part of its material form. This devel- 
oping practical understanding in professional practice is counter to arguments that 
have circulated previously within media theory. For instance, New Media: A Critical 
Introduction (Lister et al. 2003: 13-21; 35-44) claims there is an essential virtuality 
to new media, with the immateriality of digital media stressed over and over again. 
However, industrial and professional expertise now challenges academic conven- 
tion by seeking to re-inscribe digital image making as a material process. Data labs 
exist and so one can deduce that data also exists. Large companies like Google and 
Microsoft position server farms within the arctic circle to take advantage of free 
cooling - the manipulation of data generates heat. There are various other charac- 
teristics to the handling of data that enhance its materiality and evidence is mount- 
ing of its materiality as a medium. In a conversation with a colourist in London’s 
Soho, I proposed that the analogy of ‘warming the photo-chemical developer, to then 
change the characteristic response of the film that was going through the developing 
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bath’. Likewise, one might also interfere with the material nature of data within its 
processes so that the look could be created within the image. This is different from 
simply ‘baking in’ a look, which then limits manipulation during post-production. 
Instead I’m describing the possibility of artistic manipulation within the traditional 
terms of operation of the cinematographer, rather than limiting the possibilities of 
others in later processes. 

In sum, at the beginning of digital video, when data was generated at low lev- 
els via proprietary formats, it was possible to regard its limited presence as being 
immaterial—now we have to think again. In wanting as much detailed information to 
be retained as possible, Directors of Photography are expressing the desire for veri- 
similitude with the real world. This attitude must also prompt the re-investigation 
of the academic trope of thinking of digital information as being immaterial. Many 
data labs have now grown up around the world to handle the tsunami of data being 
generated by the attitude of DPs towards their craft. A major argument for the digi- 
tal as being material resides in the fact that people are employed in its handling; like 


any other material, its commodification is a sign of its existence. 


The Technological Horizon of the Image 


Film was developed with the aid of two seminal technologies preceding it. Like all 
technologies the first had several inventors engaged in its development: Alexander 
Parkes investigated Cellulose Nitrate and introduced ‘Parksine’ in 1861 and later 
an artificial dental material was introduced by the Albany Dental Plate Company in 
1870, invented by John Hyatt. Both lead to Celluloid. The other technology was the 
introduction of ‘intermittent mechanisms’ with the first functioning sewing machine 
in 1830. Barthelemy Thimonnier was nearly lynched by a mob of tailors who saw 
his invention as something that would lose them work. Nevertheless, together these 
inventions enabled cinema to come into being. 

Early frame rates (12, 14, 16, 18fps) were too slow to generate the sensation of 
smooth motion, so were then enhanced in the projector by introducing a revolving 
shutter with 4 sections—two were clear, two were covered—so each frame of film 
was displayed twice per second. Today we produce 48 (or 72 flashes if there are 
three open sections) if the film runs at 24 fps. Television emulated this flashing of 
the image by using interlaced technology that split one frame into two fields. Now 
with increased computational capability, Digital Cinematography can shoot high 
frame rates and experiments are underway with viewing images at higher frame 


rates and Higher Dynamic Range. 
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Film has always had a higher dynamic range than digital video and also digi- 
tal cinematographic equipment—but this is now changing as DC equipment is now 
around 10-14 stops of latitude, thus coming close to matching photo-chemical film. 
But Higher Dynamic Range images are now being produced on still cameras. When 
the two exposures are combined (captured in two different frames at the same 
moment), around 18-20 stops of dynamic range are produced. One problem with 
moving image HDR production is that most systems capture each exposure within 
the HDR image one after the other. This means that the two different exposures 
are time-displaced and therefore if there’s movement in the frame, blurring occurs 
which will play hell with Greenscreen. However there are now cameras that capture 
three exposures at once using a beam splitter. 

The other principle issue in HDR research is that all contemporary displays use 
a mechanism that only displays two to three orders of magnitude. This is currently 
being dealt with by joining both LED and LCD display processes so that newer HDR 
exhibition systems will display images that have the same dynamic range as the five 
instantaneous orders of magnitude of the eye brain system. 

The resolution of film is said to be 4k but requires scanning into the digital realm 
at 6k (so that it has ‘headroom’). When you previously saw a film image in a cinema it 
would often have been at 1k resolution as the processes that have been used to cre- 
ate that image have slowly but surely decreased its resolution (interneg, interpos, 
release prints, the state of older 35mm projectors etc). Therefore the average image 
captured digitally and then displayed on a 2k cinema projector is 4 times the resolu- 
tion of film display. Digital Cinema resolution is increasing month by month, year by 
year, as regards Moore’s Law. 

All in all, we are now witnessing a tsunami of development in many different 
areas with regards to motion imaging. In another example, researchers at the 
University of Bristol in the Departments of Experimental Psychology and Signal 
Processing have been conducting experiments to discover differences in immer- 
sion between 2D and 3D imaging. These tests have lead to the evaluation of motion 
stereographic imaging as only having seven per cent more immersive capability 
than 2D images of the same frame rate and the same resolution. So the project we 
humans are collectively now formulating is in fact the calibration of motion imaging 
capture and display, as regards frame rate, dynamic range and resolution so that 
it hits the ‘sweet spot’ in human eye/brain physiology. We are now in search of the 
‘perfect picture’. 

One fly in the ointment has been pointed out, albeit mischievously, by Mitch 
Mitchell (2011: ‘Verbatim History’), Head of Imaging and Archive Services at 
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Technicolor London, who argues with a degree of the pedantry born of innovatory 
thought, that 24 frames per second is more appropriate than much higher frame 
rates because it allows time between frames to reflect on the emotional and mean- 
ingful elements within the frame, thus enabling the generation of significance and 
wonder in the human brain. Mitchell’s perverse argument is derived from Stephen 
Poster, the DP who shot Donnie Darko (Richard Kelly, USA, 2001). Poster argues that 
since each frame is being flashed for 50% of the viewing time, in a two-hour film, 
one hour is effectively spent in the dark—thus allowing for plenty of time to think 
about what you’re watching. The serious point of the joke is that if you increase the 
frame rate, you won't need to flash into black all the time for physiological reasons, 


so consequently all reflection time will be lost and the romance of film will die. 


Artists and resolution 


In my own practice I have often been inspired by the simple act of making work with 
such wonderful technology. This technology functions faster than the eye or mind. 
Even analogue video takes one 64 millionth of a second to ‘write’ a line. 

‘Duration is to consciousness as light is to the eye’ (Bill Viola, quoted in 
Youngblood 2003). In this statement Viola is proposing that the presence of light is 
what caused the eye to evolve, and in turn, that consciousness evolved to deal with 
events that had duration. He is proposing that in a medium where time is an essen- 
tial factor, waiting reveals so much more. 

Viola’s roots lie in both the symbolism of Renaissance painting and the Buddhist 
proposition of Dependant Origination—that everything can only arise in relation 
to everything else. My own roots grew out of the moment that I realized that all 
things record an image through duration: from a lowly rock which, if left shadowed 
long enough, records the shadow of an image; to paper that has a leaf left on it in 
bright sunlight; to celluloid that holds a coating; to tubes, chips and sensors that 
react to light. 


Early Days 


My first encounter with videotape was in 1976 with 2-inch analogue quadruplex. 
One took a razor blade and cut it, just like film, then spliced it together to make an 
edit. Then re-recording came along, and we set up machines to record the next bit 
of video in line—thus creating an edit and also consequently, image deterioration. 
Around 1982 I was managing a video facility in Soho called Videomakers. The 


owner of the studio was the son of an electronics inventor and watched while we 
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tried to accomplish a simple dissolve between one image to another for a piece of 
work I was making. Unable to contain his excitement, he told us that his father had 
successfully harnessed a computer to ‘revolve’ a still image. He explained that with 
a little bit of development the image could be refreshed twelve times per second—so, 
by doubling and then interlacing, by splitting the image into odd and even lines, a 
whole second of video and henceforth a moving TV image could be revolved. In this 
case, through a sole inventor and not a corporation, we groped our way through the 
late analogue age and into the early neo-digital. Our main concern was how to adjust 
our thinking processes to cope with the new paradigm: to the fact that with a digital 
event, one had something that could be infinitely manipulated. One could therefore 
systematize the process—thus giving rise to ‘the operations’ as Lev Manovich (2002) 
has termed them. 

Every video artist has enjoyed the accidents that have come about through stress- 
ing the parameters of low definition equipment. However, Digital Cinematography 
offers a different kind of unveiling of form. The major difference is that image cap- 
ture can be achieved without necessarily stressing the media. This then prompts 
questions about the aesthetics of Digital Cinematography. Given that a primary 
ingredient of the artist’s palette is to find surprises within the medium itself, what 
new strategies can the artist or practitioner use to unveil a deeper insight into con- 
tent? Though McLuhan tells us this should not be so, could the message Digital 
Cinematography delivers be the beginnings of transparency? 

To return to Viola: ‘Duration is to consciousness as light is to the eye.’ But High 
Definition can deliver not just duration, but articulation. So we might now remem- 
ber how increased resolutions could affect how and what we see and therefore re- 
formulate his observation like this: ‘Definition is to consciousness—as luminosity is 


to the eye.’ 


Art and compression 


In 1987, John Wyver carried Walter Benjamin’s 1936 ideas forward, with the help 
of Jean Baudrillard and Paul Virilio, in his programme L'objet d’art a l'age electron- 
ique, broadcast on the satellite station La Sept. He asked: ‘Can a reproduction carry 
any of the authenticity of the original?’. At that time the world was concerned with 
analogue representations, which decay in their passage from copy to copy, from 
medium to medium. If one proceeded with digital compression using Fourier’s ear- 
lier mathematics, then Benjamin’s question might unveil a buried insight: To copy is 


to decrease. With digital copying this might still ring true—not only because things 
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are changed and lessened in the act of copying—but because there is a sense in 
which the representation itself is simply a ‘Borg’, a copy without feeling, without the 
‘true’ sense of the original. 

Over twenty years later, the spirit of the question still stands. Where is meaning, 
significance and value in the digital domain, given that the medium of reproduction 
and the medium of origination reside together in the same realm? Has the idea that 
things can be ‘derivative’ become defunct? Is not everything both derivative and 


original at the same time? Is the idea of the ‘original’ now anachronistic? 


Technologies, Aesthetics and Art Converging 


As there is a blurring of the lines between form and content, so there is also a blur- 
ring of the lines between software, hardware and that nether region of firmware, 
which tells hardware to be something—rather than do something. Now, through 
this blurring, and through a combination of the use of the web and digital media, a 
new kind of aesthetic is becoming available, in what has been termed convergence. 
Herman Hesse predicted post-modernism and its bastard digital child ‘Convergence’ 
in his 1943 work The Glass Bead Game. In the game itself, one might take a bar 
of Mozart and place it next to a brushstroke by Matisse, a line of poetry by Omar 
Khayyam and a silk screen by Warhol and so create a new work of art. Here, deriva- 
tion is all; in fact it has been canonized. Hesse proposes the notion that authenticity 
is not only present in the copy but that the two are one and the same—that the art- 
work’s weight accumulates with the weight of the addition of other copies and their 
imbued authenticity and all combine together into new, authentic works of art. In 
pursuit of such an aesthetic conglomerate, the actions of the new technologies and 


the way the technology is being innovated has itself become a developing aesthetic. 


Resolution/Revolution 


To return to where I began, on 31 August 2007, when Jim Jannard and Red deliv- 
ered their first complement of twenty-five Red cameras to a selected few, they set 
the world alight with their offer of cheap and high level wavelet technology (and 
made it available faster than any previous technological advance of this order). The 
introduction of 4k was a moment of industrial re-organization. This new technol- 
ogy allowed people who previously would not have had the opportunity to enter into 
the industry. This resulted in a shift in the industrial hierarchy, one that is part of 
a cyclical phenomenon that comes in waves about every five years. Overtly it looks 


like a change in technology; covertly it’s also a change in employment functions. 
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In parallel to such events, there is the ongoing development of User Generated 
Technology, coming out of an individualist trend that has somehow remained alive 
through late capitalism. For example, in the early 2000s Jeff Krienes, a Director of 
Photography in California, was experimenting with a friend from Thomson Grass 
Valley on a prototype HD Camera. They had become fed up with the slowing of tech- 
nical innovation emerging from the big corporations; so they tried to create a cam- 
era that fulfilled not only their needs, but also their aspirations. They made an alu- 
minium case that contained some electronics and a few chips, had a fitting on the 
front to take a 35mm lens and, on top, the stripped down carcasses of twenty early 
iPods to RAID record the high data output. This camera had nearly the same speci- 
fications of the Red Camera. Though Red may look like the trailblazers, they are 
in fact the inheritors of a User-Generated, YouTube-like attitude to the production 
of technology. 

In sum, in contrast to the early sole inventors in Fourier’s time, it may appear 
that we have just been through a long period where corporations, from the late 
analogue to the current digital cinematographic age have controlled the means of 
innovation. Yet as the digital reveals its nature it would seem that there is indeed 
open access to high-level technical innovation for the individual. There are recent 
incidents of companies with only one or two employees (such as the Digital Bolex 
company, which produces a 2k camera, which has already been brought to market). 
This apparent individual engagement with technology is a hallmark of our era, 
and this trend is currently centring on the production of High Definition or Digital 
Cinematographic technology. 

The commonality of information available through the web is also allowing a 
commonality of aspiration so that the User, and now the Doer, is also the Maker 
and the Knower of their own world. As we make the transition between old and new 
states, a fluttering is occurring, a switching between the two states in the suspen- 
sion of our disbelief. Through these changes, the definition of the self is expanding— 
the idea of what an individual is being re-defined as it is being up-rezzed to a higher 


level of definition. 


Postscript 


Though this chapter has its own specificities, it should be read within the growing 
understanding of what the digital realm is becoming. This can be aided by viewing a 
series of online interviews entitled ‘A Verbatim History of the Aesthetics, Technology 
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and Techniques of Digital Cinematography’, which can be found at: www.visual- 
fields.co.uk/indexHDresource.htm 

Film history between 1890 and 1915 is fairly devoid of verbatim reports from the 
practitioners and designers of the medium. It seemed to me that the development 
of HD would occur in a mirror period 100 years later between 1990 and 2015 and 
I wanted to make sure that this absence of original voices did not happen again. 
This resulted in the ‘Verbatim History’ project that was part funded by an AHRC 
Knowledge Transfer Fellowship, and also supported by the University of Bristol, Arts 
Council England, Watershed Arts Centre and South West Screen. 

Whilst making this ongoing resource I have become aware of various disabling 
predilections within current digital thinking. For instance: ideas around conver- 
gent and pervasive media have been developed with a low resolution approach to 
the world partially due to the available technology which placed a lower demand 
on the computational resources that were available. Of course higher resolutions 
demand higher processing and rendering levels, and in the beginning people were 
more interested in enabling the idea of pervasive media to function to be too wor- 
ried about the level at which it worked. 

Like notions of remediation within incoming media that seek to describe these 
changes in terms of the medium that is already existent, early forms of technology 
within a medium are less efficient and less powerful than later versions and there- 
fore aspiration tends toward lower expectations. Yet upon looking deeper at the 
notion of the convergent, one can see that it points toward a time when higher reso- 
lutions will be required—to a time where the idea of the convergent will be replaced 
by the integrative. If you are living within a world where gesture can stimulate digi- 
tal events you will want to make a small gesture with a hand rather than expend 
energy on waving your whole arm around. This is simply a question of resolution, 
and that in turn is a question of processing, compression and latency. 

The ‘Verbatim History’, though ostensibly concerned with ideas of image resolu- 
tion, is actually about the development of the digital domain in general through the 


viewpoint of motion image creation. 


Current Research (2014) 


At University of the West of England, the center for Centre for Moving Image 
Research (C M I R) will be examining developments in moving image technologies. 
My own current research strategy now centres on human physiological specificity 


and I’ve been working, in collaboration with University of Bristol and BBC Research 


HD Aesthetics and Digital Cinematography 81 


and Development, on extending my ear- Human Overall Luminance Vision Range 
lier investigation into the immersive (14 orders of magnitude) 

qualities of the image through increases 

of resolution, by combining the proper- (log cd/m?) Starlight Moonlight Indoor Lighting Sunlight 
ties of higher frame rate, higher resolu- -6 -4 -2 0 2 4 

tion and higher dynamic range images. i ae te 
Luminance Vision Range ¢—————_> 


Today's Display Technology q+ 


In November 2012 we completed the first 
ever test shoot for this level of motion 
image production at 50 and 200 frames University of Bristol Display ¢——_________ 
per Second thes lates’ Capture- projec? Figure 1. Human Overall Luminance Vision Range. 
took place in Bristol in April 2013, the 
results of which were published in a BBC White Paper at IBC in September 2013 
entitled ‘The Production of High Dynamic Range Video’, by Marc Price, David Bull, 
Terry Flaxton, Stephen Hinde, Richard Salmon, Alia Sheikh, Graham Thomas and 
Aaron Zhang. 

If you look at figure 1 it shows that the human eye/brain pathway uses 5 out of 
a 14 order of magnitude scale, sliding this instantaneous facility up and down the 
scale to deal with starlight at one end and desert sun at the other. All contemporary 
displays only currently show between 2-3 orders of this scale, but we now have a 
new prototype which displays across 5 orders and the BBC in turn have created 
a 200 frame per second projection system. By combining variants of frame rate, 
resolution and dynamic range, we should be able to effectively produce ‘the perfect 
picture’ by then calibrating these functions to produce a combination that best reso- 
nates with our eye/brain pathway—and therefore conscious awareness. The propo- 
sition is that if we can manipulate all the factors of the construction of the digital 


image then conscious immersion may follow. 


References 


Graps, A. L. 1995. ‘An Introduction to Wavelets.’ IEEE Computational Sciences and 
Engineering, 2 (2, Summer): 50-61. Accessed 29 November, 2010. http://www. 
amara.com/IEEEwave/IEEEwavelet.html. 


Lister, M. et al. 2003. New Media: A Critical Introduction. London; New York: 
Routledge. 


Manovich, L. 2002. The Language of New Media. Cambridge, Mass.: MIT Press. 


Mitchell, Mitch. 2011. ‘Interview with Mitch Mitchell.’ Accessed 25 September, 
2013. http://www.visualfields.co.uk/mitchmitchell. 


82 Terry Flaxton 


Moore, Gordon. 1965. ‘Cramming More Components onto Integrated Circuits.’ 
Electronics Magazine 38 (8) (19 April): 114-17. 


———. 1975. ‘Progress in Digital Integrated Electronics.’ IEEE, IEDM Tech Digest 
21: 11-13. 


Stump, D., and S. Billups. 2007-12. ‘A Verbatim History of the Aesthetics, 
Technology and Techniques of Digital Cinematography.’ Accessed 29 November, 
2010. http://www.visualfields.co.uk/KTV. htm. 


Youngblood, G. 2003. ‘Metaphysical Structuralism.’ The Vasulka Archive. 
Accessed 29 November, 2010. http://www.vasulka.org/archive/Publications/ 
FormattedPublications/Viola.pdf 


CHAPTER 4 


What is Digital Light? 


Stephen Jones 


The first question that arises is what in the heavens or on earth is ‘digital light’? 
What do we mean by this phrase? Because I don’t know. There’s no such thing as 
‘digital light’ as far as I can figure it out. There’s just light. We all (those of us that 
can see) know by experience what light is. It is that which we see, and the means 
by which we see. Its variations, i.e. its differences, the qualities (accidents) that it 
acquires from its sources and through its absorptions and reflections, provide us 
with the information (sense-data or data) that we use to experience the visually sen- 
sible world. That is, these variations mean that it is for us a prime carrier of informa- 
tion about the world. 

And in this modern world, light is actually more directly a carrier of informa- 
tion; digital information, through the optic fibres that will become, for example in 
Australia, the National Broadband Network. However what travels along that pipe is 
light that we do not see—what we gather from that form of light is information about 
all sorts of things from finances to messages of love to pictures of anything and 
everything. Perhaps here is our first hint. 

Now, let’s suppose that we are discussing light that can be seen, how then can 
we Call it digital? We tend to intuit that light is a continuous variation in colour or 
brightness such as we see in the shading of objects or in the transition from bright to 
dark that comes with, say, firelight. But the term digital implies that we are looking 
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at discrete countable objects and when thinking of light we might have to say that it 
is something that can be turned on and off, and we wouldn’t be far wrong here. Thus 
we might think about two kinds of light, the one being analogue light and the other 
this mysterious digital light. Moreover, we might further think of a third; the light 
in an optic fibre, which is coherent (i.e., not scattered)—enabling photonics, rather 
than electronics, and used for data or signalling. 

The first, what I have for convenience termed ‘analogue light’, raises the ques- 
tion of the physical or quantum nature of light and the second, digital light, raises 
the question of the digital or the countable or discrete. But ultimately both concepts 
imply some kind of collection of discrete objects. The quantum, at the bottom end of 
things, consists of discretised packages, and that’s what a quantum actually means. 
These packages appear to be particulate and, by implication, digital, even though 
the resolution is so extremely high it’s completely beyond us to see it as anything 
other than a continuous, analogue process. 

The quantum of light is the photon, a small probably massless entity gathered 
from matter/energy that mediates the electromagnetic force. The best description 
I’ve found so far comes from Wikipedia: viz., photons are ‘chunked ripples in a field, 
or “excitations”, that “look like” particles’ (Wikipedia s.v.; emphasis added). That is, 
a photon can be considered as a discretely located (within the limits of Heisenberg’s 
‘uncertainty’) ripple of greater amplitude within the electromagnetic field. Thus one 
can speak of particles and waves in the same breath, and it is these ‘field quanta’ that 
accord light the capacity to behave as particles in the classical Newtonian manner. 

Newton thought of light as being particulate because it travels in straight lines 
and it is blocked by opaque objects. He placed a prism of glass in the path of a thin 
‘pencil’ or beam of white light and he saw that it divided up into a spectrum of 
colours, recognising that white light is actually made up of a mix of different colours 
(Newton 1730). Because the ‘particles’ of white light are of different frequencies, 
according to their colour, they have different energies and so, as they cross the 
boundary from air to the prism, their speed of transmission differs due to the ‘opti- 
cal density’ of the glass. The product of the optical density of the glass and the fre- 
quencies of the light (which is the ‘refractive index’ at that frequency) causes a dif- 
ferential dispersion of the light and produces the spectrum of colours that one sees 
projected onto some surface (see The Physics Classroom n.d.). 

The notion that light also exhibits or possesses wavelike properties was argued 
by others, including Descartes, Hooke and Huygens, in the seventeenth century. 
This was due to observations that rays of light are not affected by crossing each 
other (Huygens 1690) and that they are not blocked by transparent solid objects. The 
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wave theory of light was experimentally proven by Thomas Young—at the beginning 
of the nineteenth century—with his ‘double-slit experiment’, in which interference 
patterns appeared when light from a single source passed through a pair of closely 
spaced slits in an opaque barrier (Young [1803] 1804). 

Nowadays, quantum theory is almost completely worked out, apart from some 
mischievous details about measurement and the role of observation. The notion of 
light, as represented by photons, is that it is somehow both a particle and a wave, 
somehow discrete and yet spread out everywhere—entangled with the cosmos— 
depending on the type of observation one makes. However, the classical notions of 
light as waves informed the work of several Australian artists and I’m now going to 
dwell briefly on the use of ‘analogue light’ by artists who have used it directly in 
their artworks. 

These begin with the colour organ, c.1912, of A. B. Hector (McFarlane 2002: 
292), and then reappear with the German emigré and sculptor Eleonore Lange 
(Batchen 1994: 49), who greatly influenced Frank Hinder (Free and Henshaw 2011; 
Jones 2011). Hinder (1906-92) went to the US in 1927 and studied in Chicago and 
New York and then worked as a lighting and set designer in Boston. On his return 
to Australia with his wife Margel in 1933 he met Lange who had brought with her 
the Anthroposophist attitude towards light as both a scientific and spiritual force. 
Hinder became quite influenced by her and spent a great deal of time over the next 
several decades trying to understand how he could represent that force of light in 
paint, and some of that work is spectacularly beautiful. Around 1967 he started 
working with light boxes with motorised lamps on metal beams, which were driven 
around inside the box behind a distorted glass screen and reflected off various mir- 
rors or through coloured gels etc. These ‘luminal kinetics’ as he called them, being 
complex, expressionist, mobile abstracts in light and fluid forms, are among the 
most interesting work done in analogue light in Australia. 

Another artist who was significantly important in this same area is Stan Ostoja- 
Kotkowski (Jones 2009; Jones 2011). He came from Poland via Germany, where he 
studied at the Kunst Akademie in Dusseldorf just after World War II. He immigrated 
to Australia at the end of 1949 and went to the Gallery School at the National Gallery 
of Victoria in 1950-51. He was essentially an abstract expressionist at that stage. He 
realised he wasn’t going to win with that in Australia at that point, so he took work 
in Leigh Creek, South Australia, as a house painter. While there he discovered how 
extraordinary the light of the Australian desert is; how intense it is and what it does 
to your perception—it really dazzles—and he began his project to paint with light. 
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After working at Leigh Creek for a year he moved to Adelaide where he began to 
play with an old TV set, fiddling around with all the controls. This is around 1960- 
62, just before Nam June Paik started to do similar things. It wasn’t long before 
Ostoja-Kotkowski began working with several engineers at Philips Research Labs in 
Adelaide and they assembled a package of oscillators for him, which, with appropri- 
ate amplifiers, could be driven into the deflection coils of a TV set. He used this col- 
lection of equipment to produce complex, almost sculptural and apparently 3-dimen- 
sional electronic images. They are not dissimilar to the complex Lissajous figures 
that Herbert Franke (1998)! and Ben Laposky (1969; see also Leavitt 1976) produced 
on an oscilloscope screen. However Ostoja-Kotkowski’s are in some ways quite dif- 
ferent from the work of Franke and Laposky since they are actually made through 
raster manipulation, becoming images in which the usual 2D surface of the screen 
is twisted and turned, producing a 3D surface in the process. You could almost say 
that Ostoja-Kotkowski and the team of engineers that he was working with at Philips 
more or less assembled the first video synthesiser, although they never packaged it 
into a single package, but this may be a contentious point (Jones 2011). 

At the very beginning of the 1970s Peter Kennedy produced a series of exhibi- 
tions using neon light, while Tim Johnson, pretty much in parallel, produced a series 
of small light sculptures that he thought of as ‘unlimiteds’. He also explored the lim- 
its of danger in a series of performances using fluorescent tubes and incandescent 
lights at Inhibodress gallery. Kennedy is one of our major conceptual artists. He was 
part of Inhibodress with Mike Parr and Tim Johnson (Cramer 1989). He had been a 
painter, right at the end of that period in painting when geometric abstraction had 
more or less reached its endgame through minimalism and the white paintings of 
Ad Reinhardt and Robert Rauschenberg and in Australia of Robert Hunter. That led 
Kennedy to a complete reappraisal of what artists were doing in making art. What 
could they do to get past this impasse that they had arrived at around 1969? At this 
time Kennedy was working as a designer at Claude Neon, who manufactured neon 
signs, and he started using the neons to make lightworks. He created an installation 
of neon tubes for Gallery A, Sydney, that exploited the intense neon colours, and 
which articulated the architecture of the gallery, effectively turning the walls—the 
white cube—into the canvass. Thus the artwork was not simply the actual objects, it 
was the whole space; the environment itself became a single artwork (Murphy 2009). 

Tim Johnson, also a member of Inhibodress and at that time a renegade from 
painting (although he returned to it after the Inhibodress period), was producing a 
body of very interesting conceptual work including a series of extraordinary perfor- 


mances where he took illuminated light bulbs on long extension leads and whirled 
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them around (Cramer 1989; Tunnicliffe Morse Code Switch 


and Ewington 2009). As happens when on 


you whirl lights around, they form contin- -Ve : = oo eas 
uous circles in your visual space because O ——$— 
the retention of the image in your retina 

holds the stimulation and makes a com- oF 

plete circle from it. Then he would let the 

lead go: Bang! And the lamp would shatter Coail 

as it hit the floor. A touch dangerous, and M oy, = 


you wouldn’t want to have been too close 


when the lamp hit the floor. Fortunately Figure 1. Morse code switch. Digital signals are switched between 
two states: ON and OFF. This can be done manually by using one’s 
‘digit’ or finger to make a connection. The result of the change of 
cally conductive, so that was not an issue. state is that current flows along the wire. 


the floor was concrete and not electri- 


All the above examples are of ana- 
logue light, and it is the means of their control that distinguishes them from the 
digital. So let’s now look into the digital. The notion of the digital comes from the 
idea that we can treat objects as being discrete and therefore countable and, since 
it was with our fingers (our digits) that we began to count them, we have been led to 
recognise that objects might be represented digitally. This notion of the discreteness 
of things is counterposed to the idea that much of what we actually look at and in 
other sensory modes hear or feel, exhibits an apparently continuous change in, say, 
strength (amplitude) or tone (frequency) and can thus be represented by some ana- 
logical value in an appropriate measuring scale. The difference between the digital 
and the analogue is not unlike the difference between the integers and the real num- 
bers. So supposing that this notion of discrete objects—and that we can count them 
with discrete whole numbers—is what we mean, then it seems to me that the ques- 
tion: ‘What is digital light?’ can be answered via the act of switching something—in 
this case light—on and off. 

In electrical terms, what digital means is simply switching between two states, 
one is on (connected), zero is off (disconnected). That’s all it means. It comes right 
back to the telegraph when it was originally developed—by one Charles Morrison of 
Renfrew, Scotland in 1753 (Mottelay 1922: 208). To send a signal what one did was to 
take a Leyden Jar, which was your battery, and two pieces of wire. You stuck one end 
of one of the wires onto the battery and its other end to something that would tell you 
that a current was flowing (often a small explosive charge), so you were literally just 
holding the wire onto, or removing it from the battery. The second piece of wire was 
simply to provide the return to the negative side of the battery. But importantly what 
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Figure 2. Manual switch. Digital signals are switched between two 
states: ON and OFF. This can be done manually by using one’s ‘digit’ 
or finger to flick a switch. The result of the change of state is that 
current flows to the lamp. 
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Figure 3. Electron Tube: Digital signals are switched between two 
states: ON and OFF. This can be done electronically by using a 
control current to permit the electron flow from the cathode (-’ve) to 
the anode (+’ve). 
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Figure 4. Transistor: Digital signals are switched between two states: 


ON and OFF. This can be done electronically by using a control 
current to permit the electron flow from the cathode (-’ve) to the 
anode (+’ve). 


is happening is that the wire is carrying 
pulses of electricity. Over the next eighty 
years, this action became more sophis- 
ticated and evolved into the Morse Key, 
which does exactly the same thing only 
now you're pressing the button with your 
finger, and of course there is that very 
useful notion of the relation of the digit 
(as finger) and the number (see fig. 1). 

So the telegraph is the first digital 
process, the first switching of signals on 
or off. Then, once the electricity supply 
was strung up through our cities, you 
have the standard light switch and when 
you switch the light on at the wall you 
are performing a digital act (see fig. 2). 
Now, of course, you can take away the 
finger and use a piece of electronics to 
do it. In the early days, with the comput- 
ers I am about to discuss, you did that 
with valves. A switching valve is a three 
electrode device or triode, with a source 
of electrons (the cathode), a control elec- 
trode, which can be switched on or off 
by an externally supplied voltage, and 
an electrode that receives the electrons 
and sends them on their way (see fig. 3). 
Here one now has electronic switching 
and this is where binary digital starts 
in terms of computing. The next stage of 
this is with transistors, which are essen- 
tially exactly the same as triode valves, 
only much smaller, and we can also con- 
trol them electronically (see fig. 4). So 
these become the basis for any kind of 
digital circuitry. 
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Since the switching has two states it can be used to represent information as 
binary data. Binary comes from the Chinese via Leibniz, circa 1700. Jesuit diplo- 
mats, guests of the Chinese imperial court, sent some documents back to Leibniz 
because he was an important savant and diplomat in that time. Among them was 
the I Ching, the Chinese prognostication system that uses a six-fold pair of symbols: 
a broken line, the Receptive (yin), and an unbroken line, the Creative (yang), which 
could be read as binary; two states. Leibniz was acquainted with combinatorics and 
recognised that these lines, the hexagrams, produced sixty-four possible results to 
the throwing of the yarrow sticks (Ryan 1996). He noted that this was a binary num- 
ber system and that such a system could be used to represent all possible numbers 
using just two symbols (Leibniz 1703: 85-89).* Given that, he went on to establish 
that all philosophical propositions and thus all philosophical arguments could be 
represented in symbols which could then be mapped onto binary numbers, allowing 
them to be tested by mathematical analysis. He proposed a binary calculator at that 
stage, but it was not implemented.? 

So binary has a long background. We can use it to count since what happens in 
numbers is that they come in cycles. If you have a cycle of ten, you count 0, 1, 2, ..., 
9, and then you have to go up to the next cycle so you add another digit. You do the 
same with binary, you just add another binary digit or ‘bit’. So binary allows you to 
represent numbers and once you can represent numbers you can represent anything, 
because you can make agreements with yourself or a larger group of people and say 
that this particular binary representation means something specific. It doesn’t mat- 
ter what it is; for example, with the evolution of the telegraph came the development 
of ways of encoding information using sequences of (binary) pulses, something that 
its original inventor, Morrison, saw as the reason for his experiments. One form of 
this became the Baudot code, which simply represented the character order in the 
alphabet as a binary number of 5-bits, and became the basis for ASCII and another, 
the Morse Code, which represented characters according to their statistical promi- 
nence in English. Thus binary could as easily represent a character value in a text 
document or a colour value in a bit-map. 

We can use binary numbers set up as digital control to switch on lights or LEDs 
much in the same way that any datum is controlled within the logical structure of a 
digital logic circuit in, for example, a computer. That is, we can digitally control loca- 
tions in memory and, thereby, store images for display. This is the crux of the mat- 
ter. This enables Digital Light. It comes about through that kind of control over the 
production of light that is made possible by some kind of digital electronic device. In 
other words, light sources that are switched, on and off, under digital control. And 
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of course this means that any kind of mechanical, electrical or electronic switch 
could be called digital because for all classes of digital logic there are two states, 
on and off. So, if you will allow me, the act of switching a switch on the wall to turn 
on the light in your lounge room is digital. The finger—the digit—touches the switch 
and causes it to change state and so the current flows and the light is turned on, or 
the circuit is broken, the current no longer flows through it and the light goes off. 
So apart from switching the light on by touching the switch, what we are probably 
really interested in here is the switching on and off of light sources by the use of 
some kind of electrically controlled two-state device in which, once either state has 
been achieved, it remembers that state until another touch of the switch, usually a 
pulse of control current, flips the switch to its opposite state. This kind of device 
is often known as a ‘flip-flop’ and it can consist in a relay or a triode valve or a 
transistor. 

Ultimately it doesn’t matter what is doing the switching, the thing about digital 
light is that the switching is binary, i.e. has two states and that it can be controlled 
by a third contact, whether that is a finger or an enable line in a logic gate. This 
gives us digitally controlled current flow. But whence comes the light? Basically any 
source of heat will produce light, we just need to be able to produce heat that radi- 
ates in the visible as well as the infra-red, and that can be switched on and off by an 
electrically controlled contact. A piece of metal can be heated and will produce light. 
So a light bulb will do, and if you remember all those early movies about the take 
over of the world by some ‘electronic brain’, then the universal sign of a working 
computer was the little tell-tale lights that winked on and off as it worked through 
addresses in its memory, with the data changing according to whatever calculation 
was being carried out. And if one really wants to think of light as being digital, and 
useful in the sense that it produces, say, images, then memory is essential. What 
memory does is to store the states of the switches in the computer or other digital 
electronic machine. The machine’s memory contains a map of its state at any instant 
and of course it can contain the details of what lights (in an array of sufficient reso- 
lution) to switch on to make an image. We can display this via an array of lamps, the 
electron beam in a CRT, or as pixel locations on an LCD, and in answer to my initial 
question: Digital Light must mean the digital control of the emission of light. 

The machine doesn’t always have to be a computer. An interesting example of 
an artwork not driven by a computer, which nevertheless utilises the storage of the 
states of an array of lamps for its appearance, is David Smith’s Kinetic Kaleidoscope. 
It was built in Sydney, Australia, over about a year from 1970-71. It consists of a set of 
twelve boards, each of which has eight flip-flops set up as a sequential shift register, 
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eight lamp-drivers switched by the state Williams-Kilburn Electrostatic Storage Tube 
of the flip-flops and the necessary con- New Data from Computer 


trol circuits to set up patterned and ran- x, Computer Data Bus & 


dom sequences to control the lamps. On Monitoring Display Beam | Pick-up 
Regeneration Amplifier 


its face are the lamps, each with a dif- a 
fusion screen to even out the filament’s 2" ini’ 
Beam OFF 


Beam Waveform 


bright spot. It could produce all kinds of 


rippling sequences, shifting blocks, back 


Line-count CPUs 
Waveforms rey 
and forth movements and other patterns. ee ar 
It also offered a degree of interaction in Vertical Sweep 
Waveform : 
that, by the use of its control panel, one Horizontal Sweep 


Waveform 
could program it to produce all kinds of 
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controlled kinetic displays (Smith 1972: Figure 5. Williams-Kilburn tube Electrostatic Storage Memory. The 
s Line-count and the Vertical Sweep waveforms control the position of 
59-61; Malina 1974: 157-59). It was the beam spot which is turned on or off by data from the computer. 


probably the first digital electronic art- This places a grid of light spots on the screen which last for long 


enough to be re-generated; and if the Pick-up plate is removed an 


work made in Australia. image can be displayed. 
So now I am going to discuss CRTs. 

The first image produced by an avowedly digital electronic device is the image that 
Tom Kilburn made for his report on the Williams-Kilburn tube, an electrostatic stor- 
age device or memory developed at the University of Manchester between 1947-48 
(Kilburn 1947) (see fig. 5).* It used what are described as ‘secondary electrons’, or 
electrons produced by stimulation of the phosphor inside the face of an oscilloscope. 
While the phosphor primarily produces photons the physical action of the electrons 
striking the phosphor also produces these secondary electrons, which leak out 
through the glass face of the tube and can be detected by a metal plate placed in 
close proximity to the glass of that face. If one taps off the charge that momentarily 
energises this metal plate and re-circulates it back into the oscilloscope, then tim- 
ing circuits produced by a computer driving this electrostatic-storage tube can tell 
the computer the state of a particular location on the tube face while refreshing 
that state so that it acts as a memory (Williams and Kilburn 1949). Effectively the 
Williams-Kilburn tube is a form of dynamic RAM but more importantly for us it car- 
ries a map of the contents of the computer’s memory and if the waveforms utilised by 
the storage tube are tapped off and used to drive another oscilloscope then we can 
actually see the contents of memory as a raster or grid (Jones 2011). This is the first 
bit-map, although its origins can be seen in the Jacquard loom and even before that 
in the Turkish carpet. But this is the first electronic version and, I argue, it produced 
the first digital light.° 


92 Stephen Jones 


Now, memory is something that is 
primarily used in computers, and com- 
puters are the primary way in which 
we control the construction and display 
of digital images (or digital light). The 
first computer to be built in Australia 
was the CSIR Mk I, at the CSIRO divi- 
sion of Radiophysics (c.1947-51) on the 
University of Sydney campus. It was 
moved to Melbourne in 1956 where it 
became CSIRAC. The logical design 


was done by Trevor Pearcey and the 
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hardware design was done by Maston 
Beard (McCann and Thorne 2000; Deane 
1997). It used a different kind of mem- 


ory known as an acoustic delay-line, but 
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it did use CRTs to display the contents 


of that memory at the grand resolu- 


tion of 20 pixels by 16 lines, and it was 
pretty obvious that if you stored bits at 
the right locations you could produce a 


Figure 6. The ‘peeing man’. Frame from an animation made by crude image. Given that, the first ‘real’ 
storing the appropriate numbers in the appropriate locations on the 


face plate of a Williams-Kilburn Tube. (i.e. not done for entertainment pur- 


poses) computer graphic produced in 

Australia was a display of the output of a 
numerical weather forecasting program written by Ditmar Jenssen for his MSc the- 
sis in meteorology (in 1957). The images are weather maps, the upper one showing 
the current day’s isobars and the lower the isobars for the next day. 

The second computer built in Australia was SILLIAC. It was built for the School 
of Physics at the University of Sydney and was used to do some important work in 
the physics of the nucleus and in astronomy. It was completed in 1956 and was oper- 
ated until 1968. It used 40 Williams-Kilburn tubes for memory (giving it 1024 40-bit 
words), and the operator could tap any word of memory and display it on a monitor- 
ing CRT with a resolution of 32 pixels by 32 lines. By filling its memory with the right 
numbers it could be made to display text. Around 1964 it was replaced by an English 
Electric KDF-9 and SILLIAC was handed over to the senior computing students who 


used it for their projects and for entertainment. Of the latter one of the things they 
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did was an animation—done by filling, Vector Display 


on a cyclical basis, the appropriate loca- 
x-axis data R x defelection 


tions in memory with the right numbers. 


It starts as an outline of a man who lifts 


Beam 


his arm and fills himself up with beer, he From i] 
Control 


Computer 


then puts his arm down and pisses it out 
(Jones 2011; see fig. 6). It is the only ani- 
mation that anyone can remember from y-axis data y defelection 


the time although there were probably 


more, but its outrageousness did make 


So the first digital light produced in 
Australia came from these two machines. Having established that one could produce 
images, the next problem was to get a greater resolution and for that, at this period 
in the history of computing, one had to use a vector (or calligraphic) display. These 
were CRT displays that functioned more like an oscilloscope and were adapted from 
the Radar display. They drew by magnetically deflecting the electron beam issu- 
ing from the cathode directly around the interior of the tube as it stimulated the 
phosphor on the inside of the face-plate to give off photons. Thus the vector display 
inscribed an image in lines onto the face of the CRT (see fig. 7).° 

Around 1967 the Basser Computing Department of the School of Physics at the 
University of Sydney, which housed SILLIAC, acquired three other machines, among 
them being a DEC PDP-8 and its attendant 338 Display, which was a 1024 by 1024 
point vector display. It could produce straight and curved lines—which were assem- 
bled from short straight lines—at high resolution and in quick succession so that it 
could build up 3D images and animations more or less in real-time. It took very little 
computing power to transfer to the display computer (and then onto the screen) the 
data about where each line or object (an assembly of connected lines) should be. Its 
limit was that it could only draw lines (vectors) and you couldn’t draw shaded areas. 
But you could produce animated drawings and record them to film. 

The PDP-8 and its display were used for projects ranging from scientific visuali- 
sation for the Aeronautical Engineering Department at the University of Sydney to 
animations and still graphics for artists and filmmakers. There was even an attempt 
to produce TV station IDs with it. The scientific visualisation (produced over 1967- 
68) was an animation (recorded to 16mm film) of the movement of air molecules 
at the very low densities encountered at the edge of space and was calculated to 
show the movement of the molecules as they impacted on each other and the leading 
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Figure 7. Vector or Calligraphic Display. The beam is pushed directly 
around the screen according to voltages imposed on the deflection 
it memorable. coils by a computer. The images are, thus, line drawings. 
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edge of an aerofoil. The programming and visualisation were done for Aeronautical 
Engineering by Doug Richardson of the Computing Department. The different 
colours of the points representing the molecules indicate the kind of impacts each 
molecule has undergone. This was the first time anybody had been able to see what 
was going on, given that you couldn’t simulate the molecular densities of the atmo- 
sphere at the edge of space with smoke flows in a wind tunnel (Vogenitz et al. 1968). 

Previously Richardson had been programming the computing department’s 
KDF-9 to draw on a plotter what are often described as ‘Spirograph’ type graph- 
ics. In 1968 Donald Brook, who was senior lecturer in sculpture at the Fine Arts 
Department, had recommended to John Bennett, the head of the Computing 
Department, that he visit the Cybernetic Serendipity exhibition at the ICA in London 
(Reichardt 1968). Bennett returned with a strong interest in the possibilities of com- 
puter art and, in 1969, he and Brook decided that they should introduce the use of 
computers to their students and to the artists at the Fine Arts Workshop (aka the 
Tin Sheds). So Bennett and Richardson, who was one of Bennett’s graduate students, 
ran a workshop and showed the students and artists of the Tin Sheds what they 
might do with the computer. However most of the artists felt that it was far too diffi- 
cult; requiring an understanding of logic and maths that was generally beyond their 
interest. Consequently Richardson was asked to find out what the artists might need 
to make it possible for them to use the computer and from the conversations he had 
he realised that he should write a program that would provide artists with real-time 
feedback of what they were drawing, rather than having to write and debug a pro- 
gram that described an image (Jones 2011). 

So Richardson looked around for a suitable way to do this and settled on Ivan 
Sutherland’s SketchPad (Sutherland 1962; 1963a; 1963b), which he re-wrote in 
PDP-8 assembly language. The project ran over 1970-72 and he then began to make 
graphics and animations that grew from his mathematical and geometric interests. 
Importantly he set up the PDP-8 facility as an open shop, inviting anyone who was 
interested to come and try it out. 

Ultimately it was Richardson and Frank Eidlitz, a commercial graphic designer, 
who worked with the system and produced most of the graphic and animation work. 
Richardson was interested in the motions of lines and shapes and the evolution of 
things over time, and his work was generally abstract and spare. His output included 
many interesting graphics and animations of 3D mathematical objects. Eidlitz had 
great skill in producing semi-abstract figurative objects with a spare line and mul- 
tiple layering. The animations were recorded to film using a camera that was con- 


trolled by the display computer. But what all of this amounts to is digitally controlled 
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light coming from the display. It is controlled by a computer, which drives the elec- 
tron beam in the CRT around the screen, and it’s that that is doing the drawing. 

Other users of Richardson’s system included the experimental video collective, 
Bush Video. Some of them were interested in the mystical symbology of the manda- 
las of Hindu Tantric geometry and similar graphic material which was also gener- 
ated on Richardson’s PDP-8 system. Along with oscilloscope drawn Lissajous fig- 
ures, the Tantra images were then used in mixdowns of various videos by Bush Video 
(Jones, 2013). 

Meanwhile in 1972, the dancer Philippa Cullen had been exploring the use of the 
theremin as an interactive interface to allow dancers to make their own music. She 
had collected a set of dances notated in Laban notation, which is a complex form of 
drawn description of the choreography of dance. But she couldn’t read the notation, 
so she asked John Bennett if the computers in the Computing Department could be 
made to show in figure animation what the notated choreography would look like. 
This was no trivial process and had never been tried before so Bennett asked one of 
the department’s lecturers, Don Herbison-Evans, to see if he could do it. Herbison- 
Evans decided that the animation of the dancing figures should be based on the use 
of ellipses to represent the major parts of the body: torso, arms, legs and head. An 
ellipse is a circle in which the centre has been divided and the, now two, foci pulled 
apart. If you use the foci at either end of the ellipse as joints then you can repre- 
sent the motion of an arm, leg, or torso (Herbison-Evans 1974). This developed into 
a program called a Numerical Utility Displaying Ellipsoid Solids (NUDES), which 
could animate the dances Cullen had brought to him (Herbison-Evans 1976). The 
project ran for some years and in 1983 one half of an animated duet was recorded 
to film which was then rear projected onto a screen in front of which the dancer 
danced the other part of the duet. The original performance occurred at the 1983 
Australian Computer Conference and in 1985 it was reconstructed using chroma-key 
techniques at the University of Sydney TV Service (Herbison-Evans 1988). 

Several interesting developments in the electronic arts followed, partly from 
Richardson’s initial computer graphic project and from on-going work by Cullen 
and her collaborators as well as from the general exploration of video. In 1975 
Richardson organised an exhibition of much of the computer art, electronic video 
and sound synthesis, interactive dance and sculpture that was being produced in 
Australia. The exhibition, called Computers and Electronics in the Arts was held in 
Canberra, over a week in March 1975, as part of Australia 75, Festival of the Creative 
Arts and Sciences. Artists who presented work included Richardson, Cullen and 
her dancers, Greg Schiemer and Phil Connor, a group led by Iain McLeod from the 
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Australian National University (ANU) 
Research School of Physical Sciences, 
Harvey Dillon, Steven Dunstan and musi- 
cians from the Melbourne-based New 
Music Centre, John Hansen, Ariel and 
Joseph El Khouri from Bush Video, and 
Stan Ostoja-Kotkowski. 

Due to the breakdown of a com- 
puter-controlled patching system built 
for Cullen by Connor and Schiemer, it 
became necessary to attach her pressure 


sensitive floors—another interactive 
device she had had built—to a DEC PDP- 


Figure 8. Images from the video screens at Australia 75: Computers 
and Electronics in the Arts. The image is produced by a computer 


reading and displaying the positions of four dancers on a set of 11 and an associated digital frame-buffer 
pressure sensitive floors. 


that could hold an image of 512 x 512 pix- 
els by 4-bits per pixel (thus R, G, B and Intensity). The frame-buffer had originally 
been built by the ANU team to assist its work in Landsat image interpretation but 
was brought along to the exhibition to show off a paint package and drawing tablet 
that they had also developed (Ellyard and Macleod 1975). The situation led to the 
PDP-11 being enlisted to read the signals from the floors through its Analogue to 
Digital Converter (ADC), and to generate an image in the frame-buffer of the recent 
history of the dancers’ movements on the floors. The RGB signals from the frame- 
buffer were then sent to John Hansen’s video synthesiser and mixed with live cam- 
era images of the performance, and the whole spontaneous collaboration is an early 
example of interactive video and the use of digital images to generate video light 
(see fig. 8). 

After the Canberra show, Hansen returned to Melbourne and continued to work 
in image synthesis, building a second video synthesiser that used one of the newly 
available Intel 8080 microprocessor chips to control the ADC and the signal patch- 
ing. The project then evolved into a computer graphics system based on the vector 
techniques of computer-aided design applications. Over the next few years he and 
his colleague, Chris Ellyard, whom he had met in Canberra, assembled a 2D drawing 
and paint program called Conjure. It originally ran on a Z80-based Matrox graph- 
ics card system and then in 1982 was ported to an IBM PC-XT. The images were 
stored as lists of vectors in the computer so that they were scale independent and 
the results could be rasterised for video display. With this approach the displayed 
graphic was a bit-map but the source was still vectors and shading was not easy 
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(see fig. 9). However, as with 3D graphics 
the vectors are where you do the work 
and then you project the results onto the 
2D plane, and the rasteriser performed 
that projection. Conjure’s main output 
formats thus became video and corpo- 
rate A-V slide production (Hansen 1998), 
and while Hansen’s work is fairly early 
we all know what can happen given all 
the developments that have taken place 
since then. One small example is the 


use of Conjure in music videos that were 


made through Heuristic Video, an inde- 


Figure 9. Image produced on the Conjure system for raster display. 
pendent and experimental post-produc-_ Calculations could be scaled to either a video raster display card or 
to a film recorder for slide production. Courtesy of Martha McKelvey. 


tion facility in Sydney. 

I want to briefly mention two other developments. One is the appearance of the 
Silicon Graphics IRIS system and the purchase of two of them by the Video Paint 
Brush Company (VPB) in Melbourne in 1983-84. They used animation software from 
Wavefront which, interestingly enough, given our topic ‘was named after the term 
which describes the front edge of a wave of light’ (Carlson n.d.) on the IRIS machines. 
The IRIS machines were rapidly followed by the Quantel Paintbox. Both systems 
were used to produce TV commercials and some computer art. Some of the opera- 
tors were Jean-Marc le Pechoux (who established the company), Sally Pryor, Andrew 
Quinn and Jon McCormack, all of whom had trained at the Swinburne Institute of 
Technology and Felicity Coonan who worked at the Sydney branch. Artists who used 
the systems at VPB included Jill Scott. 

The other development is Tom Ellard’s work with the Amiga 1000. His band 
Severed Heads was established in 1979 and he worked with some of the very 
early domestic computers including the Commodore 64. Around 1983 he teamed 
up with the author to produce video clips for the band and in 1986 began to use 
the Commodore Amiga 1000 in that project (Ellard 1998). The Amiga was an espe- 
cially important domestic level computer because of its innovative architecture. 
Commodore developed two special purpose integrated circuits and the bus archi- 
tecture placed the image memory in the same address structure as the microproces- 
sor’s memory. With the aid of the Agnes address generator chip, the Denise graphics 
chip had direct access to the image memory and this meant that getting a reasonable 


resolution graphic animation out of the machine could be done in real-time, making 
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Figure 10. Image produced on the Amiga 1000 and the Conjure at 
Heuristic Video. Courtesy of Tom Ellard. 


it excellent for use in video animation 
and titling, and making clips at Heuristic 
Video for Severed Heads (sometimes in 
conjunction with their Conjure system). 
The software that Ellard initially used 
was D-Paint II and this gave him the 
facility to create some quite complex 2D 
modelling with the capacity to translate 
and rotate images around the screen 
(see fig. 10). Together we produced a con- 
siderable number of video clips using all 
sorts of devices ranging from the second 
of my video synthesisers to a Richmond 
Hill vision mixing desk with upstream 
re-entry, the Conjure system and of 
course the Amiga 1000 which was later 


replaced by an Amiga 2000. It wasn’t important in the Severed Heads work what 


kind of machine, analogue or digital, produced the visuals just that they worked 


with the ideas and melodic structure of the music that Ellard recorded. 


So finally, in answer to my opening question: What is Digital Light? I argue that it 


is simply image-producing light that is organised by and controlled through the use 


of digital electronics, particularly computers. The key idea is the digital switch and 


its potential to function as a storage device in which a signal controls the switching 


on of a light source. 
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Notes 


1. For images see ‘Translab[4]: Algorithm & Code / VI. Visual Aesthetics In Early 
Computing (1950-80).’ http://translab.burundi.sk/code/vzx/. 


2. Leibniz comments: ‘But in lieu of the progression of tens in tens, I have employed for 
several years the simplest progression of all, which goes by two in twos, having found 
that it offers a basis for the perfection of the science of Numbers. In this manner I do 
not use characters other than 0 and 1, and then on reaching two, I recommence. This 
is why two is written as 10, and two times two or four as 100, and two times four or 
eight as 1000, and two times eight or sixteen as 10000, and so on’ (quoted in Gerhardt 
1962: 224; translated by Stephen Jones). 


3. Leibniz’s discussion of the binary in philosophy and the binary calculator is covered in 
Leibniz (1679) and mentioned in Swetz (2003). 


4. The Williams-Kilburn tube memory storage device was used in the first ‘stored- 
program’ computer, which was being built at the University of Manchester under the 
direction of Max Newman and called the Small-Scale Experimental Machine (SSEM). 
It became the Manchester Baby for which Alan Turing wrote many of the programs 
and grew into the Manchester Mark 1 which was commercialised as the Ferranti Mark 
1*. From that follows the history of computing. 


5. Oddly enough television in its original form was also digital but we’ll leave that to 
another discussion. 


6. We should note that the kind of display we call a TV or ‘raster’ monitor is in fact just a 
very carefully controlled vector display in which the image is embedded as changes in 
the amount of current flowing from the cathode onto the phosphor. 
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CHAPTER 5 


Lillian Schwartz and Digital Art at 
Bell Laboratories, 1965-1984! 


Carolyn L. Kane 


That digital computing was ever used to produce luminous colours in something 
called ‘computer art’, and later ‘digital imaging’ or ‘graphic design’, was never fore- 
seen in the blueprints of the former number-crunching and statistic-processing 
devices. And yet, due to unexpected collaborations between artists, programmers, 
and computer scientists, places like AT&T’s Bell Laboratories produced a prolific 
number of innovative digital art and experimental colour systems between 1965 and 
1984. However, due to government regulation, at Bell Labs this work was in cer- 
tain cases hidden from public view and at times, denied altogether. Paradoxically, 
in 1984, when AT&T lifted its restrictions on creative work not related to telephone 
technologies, the atmosphere had changed so dramatically that despite the relaxed 
regulation, cutting-edge projects had already been abandoned. This chapter focuses 
on the digital art made at Bell Labs between 1965 and 1984, including the strug- 
gles encountered in interdisciplinary collaborations—between visionary artists like 
Lillian Schwartz and computer scientists like Ken Knowlton and A. Michael Noll— 
and the challenge to use new computing technology to make experimental art dur- 


ing this unique time period, only now being restored to the history of new media art. 
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Bell Telephone Laboratories 


Since 1899, AT&T (formerly Bell Telephone Laboratories) had been a shareholder- 
owned public utility service. AT&T made an agreement with the government to con- 
nect independent telephone companies to its network and refrain from competitive 
or commercial endeavours. However, in 1949 an anti-trust suit was filed against 
AT&T. This led to a 1956 consent decree between AT&T and the Department of 
Justice whereby AT&T agreed more explicitly to ‘restrict its activities to the regu- 
lated business of the national telephone system and government work’ (AT&T 2005). 
While this decree stipulated that AT&T (at the time still known as Bell Telephone 
Laboratories) limit its research to telephone communications, the company was ulti- 
mately shielded from market pressures which, on the level of research, amounted 
to freedom in research endeavours. In the words of Mervin Kelly, one of many open- 
minded and innovative presidents at Bell Labs during this era, the laboratory was 
‘an institute of creative technology’. Or, in the words of Bell Labs’ then Acoustical 
and Behavioral Research Center Director, Max Mathews, ‘We had a freedom that 
few places had. Unlike at universities, where everyone seems to be competing with 
each other for scarce resources, we could cooperate. It made it a very pleasant place 
to work as well as richly productive’ (Kurcewicz). 

This openness and freedom from market pressures, or ‘golden era’ at the Labs 
as computer scientist A. Michael Noll refers to it, lasted until 1984 when a second 
antitrust suit was settled wherein AT&T agreed to give up its monopoly on the tel- 
ephone systems and compete in the marketplace with other communications compa- 
nies. After this juncture, ‘the Bell System was dead’. AT&T and seven regional Bell 
operating companies (the RBOCs) replaced it. In exchange, the U.S. Department of 
Justice agreed to lift the constraints of the 1956 decree, thus allowing research at 
the Labs to be conducted in areas not restricted to the telephone, such as emerging 
media (Kurcewicz). At the same time, because AT&T now had to compete commer- 
cially, profit became a primary goal. As a result, experimental musician and com- 
poser Laurie Spiegel, who worked at the labs in the 1960s and 1970s, explains, “... 
a lot of pure research with questionable long-term economic benefit went by the 
wayside in favor of things that would bring in revenue ... [the Labs] had to sell stock 
and compete with each in the market and fund their own research...’. Thus many of 
the visionary pioneers left, such as Michael Noll and Ken Knowlton, and new people 
came in who were ‘not the self-motivated type’ as before, but who could instead ‘be 
assigned to a project that management thought was a good thing to do’. After 1984, 


just when research into emerging media forms was legitimated, very few visionaries 
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were left to push it in new and creative directions. Even according to AT&T, their 
‘top 10 innovations’ were made prior to 1984. Under these conditions, between 1956 
and 1984, researchers at the Labs enjoyed a great amount of freedom and leeway in 
the activities and projects they chose to pursue (Kurcewicz; Kane 2008a). 

Furthermore, the end of World War II brought fresh talent, new technologies 
and a sense of future optimism to the United States and to Bell Labs in particular. 
During this time a prolific amount of innovative experimentation was conducted in 
a relatively open environment, laying the foundation for ‘computer art’, or simply, 
what has become known as new media aesthetics, a project that began ‘on the side’ 
of the official research projects. As Ken Knowlton describes it, ‘practitioners’ at the 
Labs were ‘tethered on long leashes if at all ... earnestly seeking enigmatic solutions 
to arcane puzzles. What happened there would have baffled millions of telephone 
subscribers who, knowingly or not, agreeably or not, supported the quiet circus’ 
(Knowlton 2005: 8). 

Many of the exciting crossovers between computing and art that began in the 
1960s can be attributed to electrical engineer Billy Kluver, then positioned at 
Bell Labs in the Communication Research Department and co-founder of E.A.T. 
(Experiments in Art and Technology), along with Fred Waldhauer (who was also on 
staff at the Labs from 1956 to 1987) and artists Robert Rauschenberg and Robert 
Whitman. For E.A.T.’s infamous 9 Evenings: Theatre and Engineering performances, 
held at Manhattan’s 69th Regiment Armory in October of 1966, Kliver set up collab- 
orations between many of the Labs’ engineers including Béla Julesz, Max Mathews, 
John Pierce, Manfred Schroeder, and experimentally minded artists John Cage, 
Merce Cunningham, Andy Warhol, Deborah Hay, and Steve Paxton. Kluver had been 
promoting artist-engineer collaborations by ‘courting downtown New York art- 
ists for some time’, Fred Turner notes, and by 1966, ‘by his own estimate [he] had 
taken perhaps a hundred artists on tours of Bell Labs’ (Hultén and Königsberg 1966; 
Turner 2008: 19). 

In sum, the surplus of post-war computing technology, influx of talent to the 
Labs, and freedom from market pressures, in conjunction with the culture of the late 
1960s presented to many the necessary ingredients for an optimism and visionary 
outlook on the future; a world of human and machine fusions characterized by an 
aesthetic of vibrant, electric colours fuelled by cybernetic systems that would lead 
to positive changes for the future of humankind. With the aid of electronic technolo- 
gies, both new and yet to be formed, many people believed that a new ecological 


consciousness would connect all of humanity in what Marshall McLuhan coined a 
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techno-tribal ‘global village’. Much of this decade’s magic and innervation constel- 
lated at Bell Laboratories, as this chapter addresses. 

At the same time, there were a few instances when this leeway and freedom was 
tested. As long as news of these nonofficial, on-the-side experimental and artistic 
pursuits with colour and visual computing did not get back to certain bureaucratic 
sectors of the Labs’ management, many employees, including several prominent 
department heads, supported and gave significant leeway to what they perceived as 
good-spirited endeavours. 

It is possible that new media technologies would have progressed in directions 
more fruitful for both Bell Labs and those involved if the creative ‘artwork’ of the 
artists and researchers of the 1960s and 1970s had been more widely known and 
acknowledged. This, however, was not the case. Their brief moment has passed, 
and today these works are only now being recognized in restorative histories and 
related media archaeologies. A fuller understanding of the conditions and chal- 
lenges that faced the creators of these unique and ephemeral productions serves 
to enrich mainstream histories of digital art and new media. In particular, in this 
chapter I focus on key computer artworks created by artist and long-time ‘resident 
visitor’ at Bell Labs, Lillian Schwartz, who worked on computer films and visual art 
at the Labs for the most part concealed from public view. 

There were occasional screenings of Schwartz’s work and some artists caught 


wind of what was going on at Bell Labs. One such artist was Salvador Dali. 


Interlude with Salvador Dali 


One day in 1970, American graphic artist, sculptor, and painter, Lillian Schwartz 
(born in 1927) answered the telephone in her New Jersey home (Schwartz and 
Schwartz 1992: 4-5; 10). The man on the other end mysteriously identified himself 
as ‘Salvador Dali’s Major’. A prank call she assumed. But then the voice told her to 
be at the St. Regis Hotel at 7:30 that evening and to ‘dress in your most beautiful 
gown’ (Schwartz LIL: Chapter V, 15-16). 

The Major found Schwartz in the lobby fifteen minutes early, accompanied by 
her son and a disguised German TV crew prepared to audiotape the exchange. She 
wore her ‘colored striped knitted gown’ and ‘long pink and purple Mylar earrings’. 
The Major (Dali’s business man) was wearing a full military outfit because Dali felt 
safer around the military. He escorted them into a darkened hall where Dali was 
seated on a throne wearing ‘black tails, a silk top-hat, and black cape lined with 


white satin’. He was surrounded by blonds in equally spectacular flowing white 
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crepe dresses. When Schwartz came in, Dali stood up, ‘adjusted his cape, twirled 
his moustache’ and walked towards her ‘with his cane pointing straight in front of 
him’. He motioned for her entourage to sit down behind her and then signalled for 
her to sit in a chair closer to his throne. He marched over to the table beside her 
chair where a man appeared with three white boxes. He talked rapidly in French 
and Spanish and the man interpreted: ‘Dali said he had received messages through 
his moustache, his antennae, that you and he would work together on four projects’ 
(Schwartz LIL: Chapter V, 17-18). 

Only one of these projects panned out, the subject of the first white box. Dali 
pushed this now open box towards her with his cane. He talked again in two lan- 
guages. The interpreter explained, ‘Dali wants you to examine the pen in the box, 
you may pick up the pen, but you must keep the cotton under it and not touch the 
pen’. Dali’s voice became louder and faster. The little man continued to decode, 
‘Turn the pen in different directions. Dali wants you to see the wonderful sparks, the 
gleaming, and the rays of light shooting off and out of the pen. He wants you to vide- 
otape this pen. He knows the results will be spectacular, magnificent bursts of light’ 
(Schwartz LIL: Chapter V, 17-18). 

Schwartz suggested that the particular candle lighting in the room was respon- 
sible for the gleam off the surface of the pen. This comment triggered Dali into vola- 
tile tirade of ‘Spanish, French, and some English’. But then he suddenly changed 
tones. The translator imparted, ‘Dali wants you to know that he urinated on this 
pen every morning for one year. The encrustation, the crystals on the pen catch the 
light like diamonds. He wants you to record this pen, to make a tape that will catch 
the brilliance of this phenomenon and give it permanence. Dali’s creation will be a 
great visual experience. It is your job to make a permanent record’ (Schwartz LIL: 
Chapter V, 17-22). 

Also that night, Dali took Schwartz upstairs to show her his ‘jewels’, a collec- 
tion of live miniature beetles and insects crawling on stones. When she realized 
that these little black dots were alive, she recalls feeling an ‘eerie sensation that 
my hand was cut open and ants were crawling out, just as in Dali’s painting’. Dali 
wanted Schwartz to create many extraordinary projects for him. However, for vari- 
ous reasons she denied the requests for the other projects, save for the video of the 
urinated pen, which, she reports, turned out better than she expected (Schwartz 
LIL: Chapter V, 20-21). 

But what was it about this relatively unknown artist that caught and sustained 
Dali’s attention? Dali’s work was from another era in art history: Modernism and 


its grand epoch of the genius artist, now threatened by the advent of computer art. 
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In contrast, Schwartz’ work, like others 
at the time, was focused on the future. 
The new art spoke to unknown alli- 
ances between humans and electronic 
machines. Many could not yet under- 
stand how or what this new breed of 
computer artists were up to, yet they at 
least recognized that there was some- 
thing powerful and visionary in their 
work. One other such person was Bell 


Labs’ engineer, Leon Harmon. 


The Machine at the 
End of the Mechanical Age 


Leon Harmon met Lillian Schwartz at the 
opening of Pontus Hultén’s 1968 land- 
mark exhibition, The Machine as Seen at 
the End of the Mechanical Age, an exhi- 
bition supported in part by Experiments 


Figure 1. Metal catalogue cover from Pontus Hultén’s 1968 landmark 


in Art and Technology (E.A.T.) and held exhibition, The Machine as Seen at the End of the Mechanical Age. 
The exhibition was in part supported by Experiments in Art and 
at the New York Museum of Modern Technology (E.A.T.) and held at New York’s The Museum of Modern 


Art from 25 November, 1968 through 9 Art from 25 November, 1968 through 9 February, 1969. 


February, 1969 (Schwartz LIL: Chapter 
V, 1; see fig. 1). 

On display at the opening night of the MoMA exhibition was Leon Harmon and 
Kenneth C. Knowlton’s important entry: Studies in Perception No. 1 (1966), dubbed 
the ‘Nude’: a 5 by 12-foot computer-generated nude made in one of the first com- 
puter graphic languages made for raster film, BEFLIX (an acronym for Bell Flicks) 
(see fig. 2). BEFLIX was the first specialized computer animation language written 
to produce mosaic composition languages, or simply, a computer language that could 
be used for pixel animation and bitmap sequences (Youngblood 1970: 246). Knowlton 
wrote BEFLIX in 1963 using a high-level set of macro-instructions or MACRO-FAP, 
FAP was the machine language native to the IBM 7094 machine that they were using 
at the time and MACRO-FAP indicated an additional ability to, as Knowlton puts 
it, ‘accept a definition of a common sequence of operations, for example, you could 


write min(a,b,c) to establish the value of the smallest of three numbers instead of 
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writing each time the required sequence of half a dozen FAP instructions’ (Knowlton 
1964; Kane 2008b; Carlson 2003a).? BEFLIX was capable of drawing straight lines 
from dots, drawing curves, copying regions, moving regions, doing a solid fill in 
specific areas, zooming in specific area, and dissolves and image transitions. After 
writing the programming language and using it to compose Studies in Perception 
No. 1, Knowlton output the piece in eight sections, using a Stromberg-Carlson 4020 
printer. At the time, each minute of output cost approximately $500 (Reichardt 1971: 
77; Kane 2008b; Youngblood 1970: 246). 

The final Studies in Perception No. 1 image consisted of many tiny electronic sym- 
bols including multiplication and division signs, transistors, zener diodes, vacuum 
tubes, resistors, tape reels, and writing crossovers used to compose 11 x 11 arrays. 
The genius of the piece was the visual effect it created wherein, when viewed close 
— — son, UD, it consisted of thousands of these 

Ho i : ane tiny black and white symbols, but when 

RE By viewed from a distance, another picture 
to come into view: a 12-foot female nude. 
Programming was complex and involved 
many tedious hours plotting numbers on 
graph paper, transferring them to punch 
cards, taking the punch cards down to 


Figure 2. Leon Harmon and Kenneth C. Knowlton. ‘Nude’, or Studies 


in Perception #1 (1967), featured in the MoMA’s Machine Show. the processor room, waiting in line, feed- 
Bell Labs demanded that the artists dissociate the work from the ing the cards through the processor, and 
Labs, until it appeared on the front cover of The New York Times, , : 

when they changed their original directive. Image courtesy of finally, returning the next day or later to 


Ken Knowlton. 


see what you got. Often referred to as 

‘blind programming’, one didn’t see what 
one had until the end of cycle, at which point, one usually saw errors and had to 
repeat the entire process. 

At the MoMA exhibition, Harmon was intrigued by Schwartz’s entry, Proxima 
Centauri (1968) (Schwartz LIL: Chapter V, 1), engineered by Dutch born Per Biorn. 
Biorn began working with artists during E.A.T.’s infamous Nine Evenings held at 
the armory in 1966 (Experiments in Art and Technology 1998: 7; Biorn; Schwartz 
1969). Proxima, unlike the nude, was a mechanical and kinetic light-based sculp- 
ture, perched on a 55” x 30” x 30” black plastic base with a white translucent dome 
on top. The guts consisted of an old singer sewing machine and proximity detector 
pads so when it was approached by a viewer, four switches turned on a motor that 
lowered the dome as it changed colour from blue to red. There was also a projec- 
tor located inside the black box which automatically alternated between eighty-one 
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abstract slides projected onto a mirror that reflected the image onto the interior 
surface of the frosted dome, also mediated through a water-filled ripple tank. The 
tank was agitated for five seconds every minute, allowing the image to appear to 
settle before moving to the next one (Biorn; Coffey 1984; Schwartz ‘Description of 
Proxima Centauri’; Schwartz ‘Mechanical to Electrical Computer’). 

Despite the complex mechanical setup, from the viewer’s perspective, the 
piece appears simple and elegant. When reassembling Proxima during my archival 
research in 2008, I found it remarkable that the complex set-up had been entirely 
concealed behind a plain black façade and white dome. Its hidden technical sophis- 
tication enhanced the careful game of hide-and-seek that it played with visitors: as 
one approaches the dome, it turned from a luminous blue into an alarmed red and 
began to sink back down into a hidden position in its base, remerging as a calm blue 
only when the viewer walked away. 

In comparison to Harmon and Knowlton’s black and white computer generated 
nude, the two pieces in the MoMA exhibition could not be more different. The for- 
mer, while it was based in complicated mathematics and a pioneering project in digi- 
tal graphics and optical perception, nonetheless consisted of static, geometric, and 
black and white monochrome characters printed on flat white paper. In contrast, 
Proxima was a kinetic sculpture, mechanically engineered, ushering out the ‘end 
of the mechanical age’ in luminous colour. Where the former was technically pro- 
gressive and computationally innovative, the latter was aesthetically in tune with 
avant-garde techniques for colour in film and multi-sensory media. On this even- 
ing of mutual fascination and intrigue, the two worlds came together. That night 
Leon Harmon invited Lillian Schwartz to visit New Jersey’s Bell Laboratories the 
following Thursday, after which she remained at the Labs for several years, work- 
ing on computer art and colour experiments in digital computing (Schwartz LIL: 
Chapter V, 4). 


Confrontations with the Labs’ ‘Computer Art’ 


As noted, as long as news of these nonofficial, on-the-side experimental and artistic 
pursuits did not get back to certain bureaucratic sectors of the Labs’ management, 
many employees, including several prominent department heads, supported and 
gave significant leeway to what they perceived as good-spirited endeavours. There 
were however a few instances when this leeway was tested. In 1968, for instance, 
when Michael Noll was working on the picture phone (a precursor to such products 


as video Skype or iChat), he accepted an invitation to use the device in a sequence 
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Figure 3. A. Michael Noll’s picture phone in phone booth, 
featured in Stanley Kubrick’s 2001: A Space Odyssey (1968), 
daringly bearing the 1968 Bell Labs logo. 


of Stanley Kubrick’s landmark film, 
2001: A Space Odyssey (1968). Despite 
the movie’s futuristic edge, when news 
of the scene got back to the Labs, ‘AT&T 
was furious’ because the public relations 
department deeply opposed the Labs 
being associated with commercial media 
(Kane 2008c; see fig. 3). 

Another instance occurred earlier, 
in 1965, when the Howard Wise Gallery 
asked Noll and Labs’ researcher and sci- 
entist Béla Julesz to hang some of their 
work, some of the first computer gen- 
erated images ever produced, in their 
upcoming art exhibition, ‘Computer- 
Generated Pictures’. Once the Labs 
caught wind of the event they ‘made 
an effort to halt the exhibit’ but it was 
too late. The Labs thus instructed Noll 
and Julesz to take out a copyright on 
the pictures in their own names so the 
‘art’ would not be associated with the 
Labs. However, the Library of Congress 
refused to grant the copyright to them 
‘since a machine had generated the 
work’ and this, the Library of Congress 
informed them, was ‘not acceptable’. 
Noll explained to the LOC that it was 
humans who programmed the machine. 
This explanation failed. In a third 
attempt he finally received the copy- 
right (Noll 1994: 41). Noll is also quick 


to note that while the piece was not issued a copyright until 1965, it was ‘actually 


made in 1962’, making it, not the computer art produced by the Germans, Noll is 


proud to report, the first work of computer art (Kane 2008c). Computer historian 


Margit Rosen suggests, however, that there is no known date or ‘birth year’ for the 


origins of computer art, as there is with photography and film, ‘however fictitious 
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such dates may be’. Nonetheless, she also observes that in 1960, Kurd Alseben with 
physicist Cord Passow drew four plotter drawings using an analogue computer at 
the German Electron-Synchotron research centre in Hamburg, throwing the origin 
date into question once again (Rosen 2011: 9). Regardless, Noll’s encounter with 
the LOC illustrates the foreignness of using computers to create ‘art’ during this 
era. And thus, while this kind of work has since become banal and naturalized in 
contemporary imaging, and to a large degree invisible (hence the reason why we no 
longer refer to such work as ‘Computer Art’, but instead as ‘Art’, ‘Design’ or ‘New 
Media Art’ at best), it is nonetheless important to observe how radically different 
this perspective on computer art was only a few decades ago where the term itself 
was rejected on multiple levels not only within certain sectors of Bell Labs, but also 
beyond it, in the art world and aesthetic practices in general.* 

Yet another example of these tensions between management and computer art, 
involved Knowlton’s and Harmon’s ‘Nude’. Originally made as a joke for one their 
colleagues and pasted to his office wall while he was away, when the public relations 
department at Bell Labs caught glimpse of the image, ‘They scowled and warned 
if you must “circulate this thing be sure that you do NOT associate the name of 
Bell Labs with it.”’ Nonetheless, shortly after the warning memo was issued, the 
nude debuted at Robert Rauschenberg’s loft for an E.A.T. press conference. The next 
morning, it appeared on the first page of the second section of The New York Times, 
which, Knowlton notes, ‘made not the slightest effort to conceal its birthplace’ 
(Knowlton 2005: 10). After the ‘Nude’s’ public debut, the Labs management sent a 
revised statement: ‘You may indeed distribute and display it, but be sure that you 
let people know that it was produced at Bell Telephone Laboratories, Inc’. Knowlton 
suggests the dramatic change in attitude was due to the ‘venerable’ status of The 
New York Times, not an acceptance of the fact that the Labs’ resources were being 
allotted to ‘computer art’ (Knowlton 2005: 11). At any rate, Knowlton had by this 
time learned to tell people that his computer art was ‘made in the research lab of 
a large, nation-wide telephone company that wishes to remain anonymous’ (Kane 
2008a; see fig. 2). 


Lillian Schwartz at Bell Labs 


While at Bell Laboratories, Lillian Schwartz worked with Kenneth Knowlton. In this 
section, I discuss two computer art pieces produced during this time: UFOs (1971) 
and Enigma (1972) (Schwartz and Schwartz 1992: 152 and 166; Rush 1999: 176-77; 
Lehmann 1972; City University of New York 1975: 4 and 6). 
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Gene Youngblood’s foundational text, Expanded Cinema (1970) defined the radi- 
cal art movements and practices emerging in the 1960s and 1970s. The art he ana- 
lyzed was synaesthetic, cosmic, colourful, and mystical, seeking to integrate com- 
puters and electronic circuits with human cognition and sense perception. While 
UFOs and Enigma are not mentioned in Youngblood’s otherwise comprehensive text, 
I show here how they nonetheless speak directly to and within this once expanding 
perceptual field. 

UFOs and Enigma accomplished several things for colour in early computing: 
the integration of colour techniques from painting and graphic design; the use of 
optical science and studies in perception in computer art; and third, an explora- 
tion of humans and machines as analogous yet distinct drawing and perceiving 
systems. UFOs begins with an upbeat, quick pace. Images of solid circles and half- 
moon graphics flash on and off the screen in red, yellow, blue, and green, alternat- 
ing between colours. Occasionally the images appear to overlap or they are overlaid 
with other computer-generated, horizontal lines. Soon enough the pace quickens to 
the psychedelic sound track created by Emmanuel Ghent and the coloured shapes 
become animated to such a rapid speed that one looses track of which colour one 
is looking at (Vogel in Schwartz LIL: Chapter VI, 7-9; Schwartz LIL: Chapter VI: 5; 
Digital Art Museum). Both luminous colour and music gain acceleration, becoming 
so intense that the image transforms into something else, something hypnotic and 
alien, but alien to what it is still unclear. 

While editing UFOs in 1971, Schwartz found an editing technique that increased 
colour saturation. She found that inserting a black frame between every four 
coloured ones helped to ‘keep the viewer’s eyes refreshed’ while the black frames 
remained ‘undetected during projection’ (Schwartz and Schwartz 1992: 114). After 
a 1972 screening of UFOs at the Whitney Museum of American Art, audience mem- 
bers reported hallucinations and headaches and in one case ‘uncrossed a case of 
chronically crossed eyes’ (Schwartz and Schwartz 1992: 115). The computer film, as 
explained in the Whitney’s 1972 press release for the ‘New American Filmmakers 
Series’, employed the computer to create a ‘nearly subliminal experience of abstract 
reality. The stroboscopic spheres in the second half... have been specifically created 
to affect the viewer’s brain rhythm and induce a mild state of alpha consciousness’ 
(Whitney Museum of American Art 1972: 8). 

The Whitney’s description is significant because it points to an important and 
often overlooked connection between the expanded cinema and human neurology. 
Alpha waves are meditative and associated with idleness, relaxation, and synchro- 


nized and coherent neural activity; oscillating within a frequency range of 8-12 
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Hertz. In contrast, beta waves are associated with normal neural activity, which 
is also to say more active mental activity and they run between 12 and 30 Hertz 
(Geiger 2003). Television’s radiant light (and by extension a computer screen) can 
induce a mild state of alpha consciousness, as observed in McLuhan’s analysis of tel- 
evision as a cool medium, or in artworks like Nam June Paik’s Zen TV (1965) (Moore 
2001: 61-63). And thus, when McLuhan claimed that television was an extension of 
our central nervous system, creating an auto-amputation or, ‘narcissistic trance’, 
neurologically, he was correct. Moreover the fact that electronic colour transmis- 
sions (regardless of content) induce a mild state of alpha consciousness further 
supports McLuhan’s dictum that the medium is in fact the message, not to mention 
Nietzsche’s claim, contra Wagner, that after 1900 aesthetics had become nothing but 
‘applied physiology’ (Nietzsche [1895] 2001: 104). 

Accordingly, UFOs asks how normative (human) perception can be expanded to 
see what is already in computation, but not yet known or visible? And thus there is 
something both foreign and welcoming about it. UFOs is disarming and strange; 
an assault on visual perception and traditional modes of cinematic viewing, using 
visual effects that were not uncommon at the time.° And this is perhaps why Bob 
Lehmann in 1972 wrote of UFOs, ‘It is strange to feel your body physically moving, 
directed only by the gravitational effect of a moving two dimentional image ... In 
addition to being creative, inventive and extremely colorful, the manipulating of the 
mind that goes on in some of the films of Schwartz and Knowlton is interesting and 
even a bit frightening’ (Lehmann 1972: 6). In 1971, critic Amos Vogel wrote that the 
‘stroboscopic effects’ in UFOs were ‘unsettling’. ‘Even more ominously’, he contin- 
ues, ‘while [its] design and action are programmed by humans, the result in any par- 
ticular sequence is neither entirely predictable ... being created at a rate faster and 
in concatenations more complex than eye and mind can follow or initiate’ (quoted 
in Schwartz LIL: Chapter VI, 7-9). UFOs, like many Op Art works at the time, per- 
ceptually trains audiences for increasingly rapid visual experiences, ones that have 
become more and more common in recent years.°® 

UFOs achieved a breakthrough in editing and effects research. It not only intro- 
duced a new style and colour technique that digital videos and commercials now 
mimic, but also, it brought colour into a medium that did not have it. This was done 
in two ways. The first was the use of optical techniques to intensify the colours 
through black frame inserts. The black frames functioned on the level of subjective 
perception, providing a temporary reprieve for the eyes, which allowed the other 
colours to stay crisp and fresh. In reference to a later but very similar computer art 


film, Googoplex, Bob Lehmann explains, ‘Schwartz and Knowlton have gone further 
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in that they activate the brain to receive this black and white film in color and also, 
apparently, alert the brain to follow a mathematical progression which I interpreted 
to be creating a new (deciphering a lost?) language’ (Lehmann 1972: 6). Indeed, a 
new ‘language’ that learns to speak or more appropriately, teaches viewers to ‘scan’ 
the logic of computation in visual form. Second, colour was brought into the work 
using colour alteration techniques common in avant-garde and experimental film 
practices (Schwartz and Schwartz 1992: 115). In order to fully appreciate the first— 
the adaptation of optical research into computer art—I will discuss the 1972 com- 
puter art film, Enigma. 

In Enigma, a series of black and white checkered, striped, and patterned squares 
flash on and off the screen. Eventually colour appears within and in between the 
black and white images. These colours are much softer than the bold and intense 
hues of UFOs, painted in calm but self-assured muted primary colours (reds, greens, 
and blues). At first the speed of the colour animation seems slower than UFOs, 
but eventually the black and white stripes begin to move fast, too fast to focus on 
any single one. 

For this piece Schwartz drew on the techniques from Polaroid’s co-founder 
Edwin H. Land (Schwartz and Schwartz 1992: 116), then giving lectures on colour 
perception at the Labs (Googolplex [1972] was also based on Land’s experiments to 
affect the viewer’s sensory perceptions) (Schwartz ‘Selected Art Films’). The goal 
was to produce an animated digital work that integrated experimental research in 
optics and simulated the perceptual effects of colour intensification (Schwartz and 
Schwartz 1992: 116). The result was Enigma: 


Enigma created the illusion of saturated colors even though it was shot 
in black and white. Dark frames and texture frames were inserted in 
a determined order in the black and white section of Enigma to pro- 
voke color ... Color replaced the black-and white sequences in the same 
order as those provoked by the black-and-white sequences, result- 
ing in the maintenance of a more saturated appearance of colors ... 
If the lines moved and intersected enough, an observer would start to 
perceive saturated colors between the lines. (Schwartz and Schwartz 
1992: 114; 116) 


While the application of this effect into computer art was new, the effect itself was 
not. As noted, inspiration for Enigma came from the research on colour percep- 
tion produced by Land, who was in turn influenced by nineteenth century physi- 


cist James Clerk Maxwell whose childhood toy—a spinning top—produced the same 
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phenomenon of optical colour mixing. 
Second, Maxwell’s colour experiments 
are in fact attributed to Goethe who, in 
his landmark Theory of Colours (1810), 
prioritized subjective colour mixing 
over Newton’s objective colour analysis 
(see fig. 4). Goethe proposed the ‘edge 
theory’ of colour, a thesis that correctly 
argued that colour is not in light, but 
in fact emerges in between black and 
white, a hypothesis that actually origi- 
nates with Aristotle, who argued that 
‘all hue is to be considered as half light, 
since it is in every case lighter than black 
and darker than white’ (Gowing 1978: 
55). In the nineteenth century, Maxwell 
and his peers, including Hermann von 
Helmholtz and Gustav Fechner, were 
inspired by Goethe’s work, as were the 
Op and light artists in the twentieth cen- 
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tury. What appealed to them in Goethe’s Figure 4. J. W. von Goethe. Diagram from Theory of Colours 


colour theory was the way in which col- 
our was seen and theorized on the edges overlaps and through edges. 

of perception, and thus visual experi- 

ence became highly subjective. Subjective perception and optical colour theories 
remained à la mode in avant-garde film and computer art throughout the 1960s and 
1970s. However, as I argue elsewhere, by the 1990s and 2000s these subjective tech- 
niques fall out of fashion.” 

The second technique used to generate colour in Enigma was accomplished 
through colour intensification, as discussed above. As colour appeared, more actual 
colour was added, accentuating the effects of the illusionary colour.® In addition to 
building on research in optics and colour perception, Enigma, like UFOs, expanded 
the perceptual field in early computer art. Much of this was possible because 
colour knowledge was brought into the world of early computer graphics (Schwartz 
‘Description of Computer Films’). Both of these computer art projects use rapid, stro- 
boscopic computer animations to generate colour in subjective perception. Enigma 
highlights how colour exists in between the (objective) screen and the (subjective) 


(1810). Goethe’s edge theory demonstrates that colour is 
not in light (as Newton argued), but in fact emerges from 
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human. Its blueprint was created during programming, editing, and production, but 
the work itself only comes to life when it is being watched; expanding perception. 

To approach colour in this way—through its material-technical and subjec- 
tive attributes—is to embrace the paradox of colour as its root. In Enigma, both 
sacred colour and synthetic colour form constitutive tensions for the work to work 
(where sacred colour tends to bear anthropocentric associations, while synthetic 
colour tends to denote artificial and machine-made colour). However, sacred must 
not be confused with anti-scientific, naive, or romantic notions of colour. Rather, 
as this work clearly demonstrates, both sacred and synthetic colours co-exist, in 
this particular historical and cultural moment of technological intrigue, along with 
a fascination with the future, and progressive social and political attitudes towards 
the human-machine consciousness. It may even be the case that it is only in these 
moments prior to a colour’s standardization and democratization that such co-exist- 
ence is visible or possible. 

Moreover, much of what this piece has to offer has yet to be realized (or prop- 
erly documented) in media art histories. For instance one of the Bell Labs’ techni- 
cal reports for Enigma notes how it was an ‘Experiment combining various filters 
to produce the spectrum of color. The sequences used to induce psychological and 
physical effects could eventually be part of a computer language and coded into the 
final programming’ (Schwartz ‘Motion Picture/Computer System’). Such a program 
(for better or worse) has yet to be seized by commercial or artistic endeavours, yet 


this kind of software could potentially provide fascinating possibilities. 


1984 


By the late 1970s and early 1980s, political conservatism and corporate ambition 
had infiltrated the Labs, bringing an end to the golden era of liberal experimenta- 
tion that took place in the 1960s and early 1970s. In 1981 Ken Knowlton wanted to 
patent a new technology that was a direct precursor to today’s text messaging sys- 
tems, but the Labs’ (now AT&T’s) public relations department answered ‘No’. They 
did not want him ‘to talk or lecture about it anymore’. When he then asked them, 
‘What is my job here?’ they couldn’t name it. They said ‘we don’t know’. At that point 
it was clear to him that things had dramatically changed and, being one of the last 
creative experimental technologists remaining, Knowlton subsequently took a job at 
the Stanford Research Institute (now SRI International) (Kane 2008b). 

Bell Labs’ temporary denial of this early experimental work in an area that is 


today not only central to ‘telephone’ communication technologies but to almost all 
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digital media, reflects the Labs’ status as a government owned monopoly, a status 
that only ended with the deregulation in the 1980s. Thus, while this protected status 
temporarily shielded the researchers from market pressures, it may have also cost 
the Labs an innovative edge in the new media industries and international notoriety. 
Bell Labs was one of several U.S. research centres that, like Xerox PARC and IBM, 
have since been overrun by market influence and changing political and economic 
tides, including the ubiquity of the personal computer, mass-produced and template- 
driven ‘shrink wrap’ software, and the rise of digital media centres and computer 
art programs in the 1980s and 1990s. Three interrelated factors explain how this 
happened. Prior to 1984, even if Bell Labs saw a future in computer graphics, the 
pressure of the government’s 1956 decree that restricted the Labs to carrying out 
only those research projects related to the telephone meant that it may have been 
unfruitful to pursue patents on these developments. (So even if this kind of work was 
being conducted at the Labs, which it was, it was best not to publicly announce this 
to taxpayers.) Second, even when those working at the Labs wanted to claim author- 
ship, if a project was too ‘artsy’ or too far beyond what could be defined as ‘tele- 
phone communication’, it was not publicized as one the Labs’ proud achievements, 
and increasingly so by the late 1960s and 1970s. And third, because these artworks 
were a product of collaborations that were sometimes unknown, even if the artists 
had wanted substantial recognition, pursuing it was (and remains to be) precari- 
ous. Finally, a number of these extensive collaborations spanned several undocu- 
mented years and at times involved participants who left little to no trace or record 
of their work. 

All of these factors explain why some pioneering works of computer art had for 
many years gone unrecognized, beyond the occasional footnote, in computer and 
media art histories. Telling the story of Lillian Schwartz and her colleagues at Bell 
Labs, within the context of the technological achievements of the 1960s and 1970s, 
continues to enrich the cultural and aesthetic histories of early computing technolo- 
gies, the role of women in these histories, the newly emerging subfield of colour 
studies in the media arts, and the major aesthetic histories and new media practices 


today. These revisionary histories, however, are only now being written. 


I thank Lillian Schwartz, Laurie Spiegel, Ken Knowlton, Michael Noll, 
and Alex Galloway at New York University. 
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Notes 


1. This chapter has been adapted and reprinted from ‘Digital Art, and Experimental Color 
Systems at Bell Laboratories, 1965--1984: Restoring Interdisciplinary Innovations 
to Media History.’ Leonardo 43, 1 (February 2010): 19, 53-58, with material that 
also appeared in Chapter 4 of Carolyn Kane, Chromatic Algorithms: Synthetic Color, 
Computer Art, and Aesthetics after Code (University of Chicago Press, 2014). 


2. Mervin was president between 1951 and 1959. James B. Fisk was president from 1959- 
1973, and William O. Baker from 1973 to 1979 (Gertner 2012: 3). 


3. BEFLIX was also used to produce the Stan VanDerBeek / Ken Knowlton Poem 
Fields artworks and was later used as the base for building TARPS, a 2Dimensional 
alphanumeric processing language. 


4. For more on the problematic reception of computer art in the art world see the 
excellent work of Grant Taylor. 
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For example, the many psychedelic and stroboscopic experiments being developed 
throughout the United States and abroad (see Geiger 2003). 


This does not refer to the stroboscopic animations, which remain challenging for 
viewers to watch. I thank Mimi White at Northwestern University for pointing this out. 


For more on this see chapters 5 through the postscript in Carolyn L. Kane, Chromatic 
Algorithms: Synthetic Color, Computer Art, and Aesthetics after Code (University of 
Chicago Press, 2014). 


This was likely done using an optical bench and animation stand, as used in some of 
the other computer films (Schwartz and Schwartz 1992: 114; 120; 153; Schwartz LIL: 
Chapter VI, 4). 
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CHAPTER 6 


Digital Photography and the Operational Archive 


Scott McQuire 


The camera has been implicated in crime scenes at least since the late nineteenth 
century. However, in the 1980s this took a novel turn, as photography found 
itself the victim in a murder apparently committed by computers. According to 
Nicholas Mirzoeff: 


After a century and a half of recording and memorializing death, pho- 
tography met its own death some time in the 1980s at the hands of com- 
puter imaging. The ability to alter a photograph digitally has undone 
the fundamental condition of photography—that something must have 
been in front of the lens when the shutter opened, even if questions 
remained as to the ‘authenticity’ of what was recorded. It is now pos- 
sible to create ‘photographs’ of scenes that never existed without the 
fakery being directly observable. ... The point is the photograph is no 
longer an index of reality. (1999: 86) 


Mirzoeff’s statement represents a familiar ambit claim from the period, amalgamat- 
ing assertions about the impending demise of photographic referentiality with con- 
cerns about the status of photographic meaning (authenticity). Well-known writers 
such as Fred Ritchin (former picture editor of The New York Times Magazine) voiced 
similar concerns about compromised journalistic ethics and the evisceration of pho- 


tography’s evidentiary value (Ritchin 1990). The suddenness of the shift in debate 
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is starkly visualized by Google’s Ngram Viewer (which allows text string search of 
Google’s database of books). If you type in the phrase ‘death of photography’, the 
graph suddenly spikes around 1990. 

Two decades later, claims about the death of photography are less fashionable. 
In part, this is undoubtedly because, as Peter Osborne (2010) argues, some aspects 
of what he calls the ontological anxiety surrounding the arrival of digital photogra- 
phy were simply misplaced. Dividing photography into phases of capture and image 
production, Osborne argues that digital capture in fact ‘retains both the causal and 
deictic aspects of photographic indexicality’ (2010: 63).1 He goes on to suggest that 
the anxiety about photography’s relation to ‘the real’ arises, at least partially, from 
the constitutive disjunction between capture and production—which, of course, was 
also characteristic of analogue photography.’ In fact, similar arguments were occa- 
sionally voiced at the highpoint of the ‘death of photography’ discourse. In his intro- 
duction to The Photographic Image In Digital Culture, Martin Lister (1995) pointed 
out the long tradition of image manipulation in chemical photography, and under- 
lined the way that the strong body of critical writing on photography emerging in 
the 1970s and 1980s had already undermined the authority of reductive claims about 
photography’s privileged relation to ‘the real’ prior to the arrival of digital cameras. 

Perceived continuities between analogue and digital imaging has led some crit- 
ics to reject the entire thesis that the digital constitutes a fundamental change in 
photographic practice (for example, see Batchen 2011). However, while this stance 
may have been a welcome corrective to the peculiar combination of marketing and 
theoretical zeal that surrounded digital photography in the dotcom 1990s, it seems 
less convincing in the present. Even accepting Osborne’s contention that digital pho- 
tography retains key attributes of indexicality, and acknowledging that instability in 
the meaning of photographic images is scarcely new, I still want to argue that the 
uptake of digital cameras has driven fundamental changes in photography. While 
these changes are directly related to widespread uptake of digital cameras, it is 
important to emphasize that they are not driven simply by changes in camera tech- 
nology. Nor, one might add, are they purely technological. Perhaps for this reason, 
they were largely absent in the first generation debates about digital photography. 
Rather than relating to the nature of the image, manifested in the demise of refer- 
entiality or the apparent loss of evidentiary value in professionally-made images, the 
transformation I am concerned with here concerns the integration of photography 
into the network milieu, and particularly the emergence of new practices for circu- 


lating and archiving everyday images.? 
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A Matter of Size 


To situate this shift in focus, consider the following trajectory. While it was well rec- 
ognized in the 1990s that digital cameras were making image capture progressively 
cheaper and easier, the implications of this transition were not fully appreciated. 
The emergence and rapid uptake of photo-sharing websites such as Flickr (estab- 
lished in 2004) was one sign.* A less evident but equally powerful signal was the mil- 
lions of personal computers that were starting to overflow with digital images. This 
is the first line of differentiation I want to mark in considering the transition from 
analogue to digital photography: dramatic growth in the volume of photographs. In 
some respects, this growth can be explained by the fact that digital capture meant 
that amateur photographers could increasingly adopt the shooting practices used 
for decades by professionals (keep shooting and select the usable images later). 
Another powerful driver was the fact that large numbers of people were now car- 
rying a ‘camera’ in many more situations than had previously been the case. It is no 
surprise to learn that the most popular ‘camera’ used for uploading images to Flickr 
at the time of writing is, in fact, the iPhone 5.° (Of course, such pragmatic explana- 
tions for the growing numbers of images captured offer little purchase in explaining 
why making and circulating photographs has become increasingly integral to con- 
temporary experience.) 

Growth in the number of digital images captured has driven the emergence of 
new practices of archiving and circulation. By August 2011, Flickr reported that it 
hosted some 6 billion images. To put this number into context, one of the world’s 
major public image archives—the US Library of Congress—held just over 12.5 mil- 
lion photographs in 2009.’ While I am not making any claim about the value of the 
content housed in the respective archives, it is worth noting that the Library of 
Congress has been operating for more than two centuries, while Flickr has amassed 
an image collection 500 times larger in a mere seven years. And, of course, Flickr’s 
photo archive is itself dwarfed by the scale and explosive growth of the image col- 
lection held by social network contemporary Facebook. While it is difficult to get 
accurate ‘official’ figures, let alone to verify them, Facebook is undoubtedly the 
biggest image archive on the Internet by a considerable factor. Various company 
blog posts track the growth in the volume of uploads, from some 850 million per 
month in mid-2009 to 2.5 billion per month in early 2010.8 By August 2011, this had 
grown to 250 million photos uploaded each day, equating to a staggering 7 billion 
per month.’ Estimates of the total image archive held by Facebook vary from the 90 
billion claimed by Facebook engineer Justin Mitchell on quora.com to the 140 billion 
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claimed by Good (2011).!° But the volume of daily uploads would indicate that this 
figure may itself nearly double in the coming year. 

Fortunately, given the brittleness and especially the volatility of such statistics, 
it is not the precise number of images but the trajectory that concerns me most here. 
There is clearly a major disjunction between the scale of traditional image archives 
and their digital counterparts. Such a disjunction poses the familiar question: does 
size matter? Is this, as philosophers once liked to ask, a change of ‘kind’ or merely 
a change of ‘degree’? Without accepting the stability of this binary, I am suggesting 
that, in this instance at least, size can function to index a fundamental change in the 
social relations of the photographic image. Moreover, rapid expansion in volume is 
near inseparable from the new modalities of storage and circulation. In this nexus, 
we are witnessing the transition of the photograph from its historical existence as a 
visual artefact where meaning is governed by an aesthetic-interpretative process of 
‘reading’ to a condition in which the image increasingly functions as a form of data 


regulated by statistical/algorithmical processes. 


Photography in context: The Problem of Meaning 


Having stated my basic argument, I now want to approach the issue from another 
direction. As I noted above, one aspect of the concern directed at digital photogra- 
phy in the 1990s was a perceived loss of ‘authenticity’. How can the photographic 
image continue to ‘guarantee’ the existence of what it shows when pixel by pixel 
manipulation allows the seamless modification of appearances beyond the threshold 
of human perception? But the disarming ease of digital modulation should not itself 
disguise the extent to which it has always been problematic to treat photographic 
appearances as self-evident. This problem has been central to long-standing argu- 
ments over photographic objectivity and the stability of photographic meaning. As 
John Berger noted three decades ago, the camera altered our relation to the world, 
first, by introducing an unprecedented temporality in its recording of the visible, 


and, second, by rendering those recorded appearances mobile: 


Photographs preserve instant appearances. Habit now protects us 
against the shock involved in such preservation. Compare the expo- 
sure time of a film with the life of the print made, and let us assume 
the print only lasts for ten years: the ratio for an average mod- 
ern photograph would be approximately 20 000 000 to 1. Perhaps 


that can serve as a reminder of the violence of the fission whereby 
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appearances are separated by the camera from their function. (Berger 
and Mohr 1982: 51) 


Separation between the moments of ‘capture’ and ‘production’, manifested as the 
gulf between the moment represented and the moment of viewing, have led to fre- 
quent demands for restoring the ‘context’ of the photograph. Loss of context is the 
fundamental danger that animates Susan Sontag’s (1977) poetic but ultimately pes- 
simistic elegy on photography.'! Other such as John Tagg and Allan Sekula empha- 
sized the importance of institutional and discursive contexts for establishing photo- 
graphic authority and shaping photographic meaning. In his influential essay on the 
archive of photographs assembled by a mining company, Sekula argued that archives 
constitute a ‘territory of images’, adding ‘not only are the pictures in archives often 
literally for sale, but their meanings are up for grabs’ (1989: 116). Back in the 1930s, 
Walter Benjamin advocated the use of captions as a means of ensuring the correct 
tendency of the otherwise promiscuous photograph. (Interestingly, this is close to 
the strategy advocated by Ritchin (2008) in a recent book, where he moves beyond 
the ‘shock of the new’ to embrace aspects of digital photography, including caption- 
ing and tagging, as means of providing context). Roland Barthes, on the other hand, 
remained far more sceptical of textual anchors, suggesting that their use was symp- 
tomatic of the general poverty of the sort of press photography which remained ina 
purely illustrative register (1983: 204).!” 

Photographic history is littered with images that have had their meaning altered 
by entry into a new setting. Digital images, with their easy availability for ‘photo- 
shopping’, are ready targets for modes of appropriation that reinforce clichéd inter- 
pretations and reductive stereotypes. And when every famous image—and many 
ordinary ones—seem to end up in the maw of advertising, it is all too easy to become 
cynical about photography in general. Against this heading, and determined to 
retain the sense of a possible political project for photography, John Berger argues: 
‘All photographs are possible contributions to history, and any photograph, under 
certain circumstances, can be used to break the monopoly which history today has 
over time’ (Berger and Mohr 1982: 109). While recognizing the inherent openness of 
the image, Berger reminds us that the determination of a ‘context’ in which a par- 
ticular image will be read is a matter of agonistic struggle rather than one resolved 
by general protocols. 

In some respects, Berger’s position reminds me of the ambiguity articulated by 
Siegfried Kracauer in his 1927 essay ‘Photography’. Kracauer (1995) memorably 
describes photography—or rather the emergence of a modern image culture based 
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on technical reproducibility—as the ‘go-for-broke game of history’. This assessment 
was premised on photography’s novel and rapacious capacity to turn the entire 
world into a stock of images—a warehouse for all of nature—coupled to its promis- 
cuous overturning of traditional categories of interpretation, such as art historical 
genres. This is the nub of Kracauer’s ‘gamble’: photography’s unprecedented ability 
to treat the existing order of things with indifference carries enormous potential to 
delegitimize established forms of social and political authority, and to allow a radi- 
cal re-assessment. However, failure to seize the historic opportunity presented by 
the transition from a traditional to a modern image culture could have dire conse- 
quences extending well beyond photography. Without radical political change, ‘the 
nature that [photography] failed to penetrate would sit down at the very table con- 
sciousness had abandoned’ (Kracauer 1995: 61). Here the ‘indifference’ of photog- 
raphy would feed the victory of commodity culture, lending its force to a spectacu- 
lar society in which images become the ultimate commodity. However, if capitalism 
were overthrown, ‘then liberated consciousness would be given an incomparable 
opportunity. Less enmeshed in natural bonds than ever before, it could prove its 
power in dealing with them. The turn to photography is the go-for-broke game of his- 
tory’ (Kracauer 1995: 61). 

While both Kracauer and Berger consistently emphasize the ambiguity of photo- 
graphic appearances, it is important to underline that this ambiguity is irreducible. 
No photograph can guarantee its own meaning and every image is susceptible to 
re-use. Providing a new context for an image may enable it to respond to Berger’s 
call for ‘breaking the monopoly of history over time’. But it is vital to recognize that 
this is never a matter of ‘restoring’ some presumably original context, as if there 
were only one. This would amount to fetishizing ‘context’ as the means for appar- 
ently controlling the polysemy of photographic meaning. Part of the ‘modernity’ 
of photography is precisely the way it foregrounds the impossibility of providing 
what Derrida (1982) terms a ‘fully saturated’ context.'* There is always a remainder, 
something left over, the possibility for re-interpretation in a new context. This possi- 
bility, which is the very condition of communication, is becoming susceptible to new 


forces in the digital present. 


Image Torrents 


If photographic meaning has always been unstable, the unmooring of images from 
their diversity of contexts—the events they represent, their conditions of capture, 


their histories of production as image files, their routes of circulation and pathways 


127 


128 Scott McQuire 


of reception—has arguably become even more complex and volatile in a culture of 
cut, paste and forward. Numerous accounts of contemporary media culture stress 
the sheer force of numbers. Todd Gitlin (2001), for instance, appeals to natural 
metaphors such as ‘torrent’ and ‘deluge’ to signal a media system out of control. 
But this is merely the flip side of the common use of similar terms as the general 
appellation for digital files accessed over the Internet. (Plug the word ‘torrent’ into 
a search engine and what appears is a list of clients such BitTorrent or sites such as 
ThePirateBay.) 

In fact, perceptions that we are subjected to an excessive number of images have 
been with us for some time. Sontag (1977) concludes On Photography with a call 
for an ‘ecology of images’, while Kracauer described the illustrated magazines of 
the 1920s as unleashing a ‘blizzard of photographs’ (1995: 58). The counterpoint to 
the more recent variants of these arguments is Clay Shirky’s (2008) pithy observa- 
tion that the issue is not information overload but ‘filter failure’. Despite his rather 
reductive techno-optimism, Shirky situates an important aspect of the transition 
from analogue to digital images. The digital image is now only partly a ‘picture’ in 
the classical sense. It is also becoming an element of a dataflow. In this context, new 
modalities for regulating meaning, and new rules for providing order in the ‘deluge’ 
of images are emerging. 

Lev Manovich’s recent work on cultural analytics (2011a, 2011b) situates some 
of the stakes at play in this shift. As particular institutional collections of digital 
images grow into massive data sets, they can become subject to new forms of analy- 
sis. The emergence of so-called ‘big data’“ in the form of image archives is itself a 
subset of the more general shift to what might be dubbed a ‘database society’. In 
June 2008, Wired ran a special issue on ‘the Petabyte age’, arguing that: ‘Our ability 
to capture, warehouse and understand massive amounts of data is changing science, 
medicine, business and technology’ (Wired 2008). Whereas Kracauer’s earlier ref- 
erence to the ‘warehousing of nature’ through photography evoked the transfer of 
industrial techniques for the production of goods onto the realm of images, here the 
transfer involves industrial techniques of computing—what UC Berkeley Professor 
Joe Hellerstein (2008) calls the ‘industrial revolution of data’—in which data is liter- 
ally warehoused in temperature controlled server farms situated in strategic (tax- 
effective, network-efficient) locations. 

‘Big data’ society carries profound implications for traditions of knowledge and 
research that extend well beyond my present concerns (Latour 2007; Savage and 
Burrows 2007; 2009). These debates do, however, help to indicate some of the ways 
that the process of constructing photographic meaning is beginning to change as 
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large-scale image sets become searchable and researchable according to a range of 
different criteria determined by software applications. In particular, the old division 
between surface and depth analysis (aggregating shallow data gained from many 
examples versus multi-layered data sourced from fewer) that animated historical 
distinctions between statistics/social sciences and hermeneutics/humanities has 
begun to lose some of its purchase in a context where ‘sampling’ is starting to give 
way to ‘whole population’ monitoring.’* ‘Reading’ an image as part of a pictorial tra- 
dition can be complemented or even replaced by processing images as data. 

It’s worth unpacking this trajectory in more detail. The concept of treating pho- 
tographic images in relation to statistics is not in itself novel. Drawing on Georg 
Simmel’s observation concerning the displacement of qualitative by quantitative 
differences in the context of the modern city, Walter Benjamin perceptively placed 
photography alongside statistics as a novel technique of ‘reality-testing’ in the mid- 
1930s. Benjamin (1999: 255-56) argued: 


Uniqueness and permanence are as closely entwined in the [artwork] 
as are transitoriness and repeatability in the [reproduction]. The strip- 
ping of the veil from the object, the destruction of the aura, is the sig- 
nature of a perception whose ‘sense of sameness in the world’ has so 
increased that, by means of reproduction, it extracts sameness even 
from what is unique. Thus is manifested in the field of perception what 
in the theoretical field is noticeable in the increasing significance of 


statistics.!© 


For Benjamin, it is the ease of reproduction coupled to the radical indifference of 
the photographic image towards what it shows, and the way this indifferent multi- 
plicity privileges exhibition value over ritual use, that underpins the general trans- 
formation of modern perception. Like statistics, photography functions as a great 
leveller of phenomena: in Kracauer’s terms, photography’s capacity to warehouse 
all of nature allows everything to be related, if only as a set of images. Paul Virilio’s 
analysis of the use of photography in aerial reconnaissance in World War I simi- 
larly foregrounds the emergence of images as an information flow capable of map- 
ping the highly mutable terrain of the frontline under high-explosive bombardment. 
Elsewhere (McQuire 2008), I have argued that the image sets generated by Charles 
Marville during his role as the photographer of the city of Paris as it underwent 
radical modernization at the hands of Haussmann also pioneered a proto-statistical 
approach to images. What links these particular instances to Benjamin’s analysis 


of the statistical ‘logic’ of technological reproducibility is that they exemplify early 
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settings in which the photographic image came to function less as a group of inde- 
pendent pictures, in which each photograph constitutes its own ‘event’, than as sets 
of relational data. Once the image begins to be conceived from the point of view of 
its membership in a series, meaning is no longer a function of singular and appar- 
ently autonomous pictures but is constituted as a relation between serial images. 
Photography becomes a means of constructing a database. 

These incipient uses of photography correspond to the formation of what Hacking 
(1990) calls ‘statistical society’, referring to the way that new modes of knowledge 
emerging in the nineteenth century contributed to the reconstitution of the social 
bond, the reconfiguration of individual identity, and the recasting of the relation of 
citizens to the state, and to each other. Photography is directly implicated in this 
transformation, not least through its formal role in constituting the official visual 
memory bank of various disciplinary institutions, including the police force, the asy- 
lum, the prison and so on. Allan Sekula (1986) has argued that, historically, two 
primary methodological axes emerged for using photographic images to define and 
regulate social differences in disciplinary society. The first corresponds to what he 
calls generalization, where photographic contingency is converted into typicality by 
making the photograph into an ‘example’ of some broader tendency. The second is a 
process of individualization which depends on a ‘machine’ (a filing system and cleri- 
cal apparatus) to retrieve a particular instance from the archive’s infinite bounds.!” 

In fact, the first approach proved more controversial and, perhaps for this rea- 
son, was adopted more rarely within science.'® There were occasional forays into 
using photographs for scientific generalization, such as Francis Galton’s (Darwin’s 
cousin and founder of eugenics) use of photographic superimposition to create sup- 
posedly generic images of social and racial ‘types’!’. But these pseudo-scientific uses 
were easily outweighed by more ‘aesthetic’ practices in which individual images, 
selected for newspapers, books, or exhibitions, were saddled with the impossible 
task of summing up an event, era or generation.” It is Sekula’s second axis of indi- 
vidualization that proliferates in concert with the other techniques of the disciplin- 
ary dispositif. Sekula’s primary example of this trajectory is Alphonse Bertillon who 
was so central to the adoption of photography in police work in the late nineteenth 
century. Bertillon’s system, in which photographic images were regulated through 
formal protocols for shooting and display, coupled to specific forms of annotation 
and mechanisms of filing and retrieval, exemplifies the process of translating indi- 
vidual instances into populations that could be statistically regulated, according to 
the innovative process that Hacking aptly describes as ‘making up people’ (1990: 
3). Despite the displacement of the Bertillon system by fingerprint identification, 
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photographic archives charged with the task of securing bodies to identities have 
continued to exert enormous and even growing power into the highly surveilled and 
increasingly securitized present. However, the ‘machine’ that Sekula referred to 
is no longer a filing cabinet but a computer, and the ‘clerical apparatus’ has been 
replaced by the new assemblage of software and a multitude of users. 

In this context, archives held by various government authorities and contem- 
porary user-generated photo archives such as those held by Facebook and Flickr 
lose their distinctiveness, as even the ‘personal images’ that were once stored in 
privately housed photo-albums can now be treated as ‘populations’ subject to pat- 
tern recognition. Growing use of automated systems to organise digital photographs 
has made explicit something that was, arguably, always an ‘affordance’ of photog- 
raphy: namely that, as the first industrialized image, photography was never simply 
an extension of the drawing and painting to which it was so often compared, but was 
the prototype of visual data. Today, as visual data is digitized and thus made increas- 
ingly susceptible to algorithmical operations, Peter Galison’s (1997) important his- 
torical distinction between ‘visual’ and ‘logical’ or counting machines is eroding, 


and photo-archives move onto an operational footing. 


The Social Protocols of Photography as Big Data 


In his response to Susan Sontag’s On photography, John Berger (1984) was moved to 
make a distinction between photographs that belong to private experience and those 
which are used publicly. Berger understands the private photograph—prototypically 
a photograph taken by one of the people viewing it, or which represents people, 
places or events familiar to its viewers—as one that is read within the context of 
the lives that thereby inform its testimony. Because the private photograph’s ‘quote 
from appearances’ is read in the context of lived experience, it does not have to func- 
tion in isolation but is drawn into a network of associations. It intersects multiple 
narratives. Looking at our own photographs or those of friends or family members is 
frequently an incitement to storytelling, an occasion for circulating and exchanging 
tales of what we felt and what others remember. Instead of being overburdened with 
demands to provide evidence, a photograph in this context can be treated lightly, a 
touchstone for recollection and recounting rather than a positive token of proof. 

In contrast, Berger argues that the contemporary public image, such as those 
seen every day in a newspaper, on the Internet, or on television, frequently arrives 
from so far outside the experience of those who view it that the gulf separating 


appearance from significance defies the capacity of imagination. This doesn’t mean 
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the image therefore has no meaning; rather, that meaning is all too easily chan- 
nelled in particular directions, heavily reliant on stereotypes. Berger memorably lik- 
ens the effect of the contemporary public photograph to the bewilderment of being 
exposed to the memory of a stranger: ‘It records an instant in which this stranger 
has shouted: Look!’ (1984: 52). Treating the world as a sequence of such bewildering 
moments corresponds to the logic of spectacle, producing the lack of specificity and 
attenuated significance for which contemporary media culture has become notori- 
ous. In this setting, the camera’s potential to redistribute cultural and experiential 
horizons is less an interruption to established political contours than the confirma- 
tion of their present order. 

Berger’s distinction between public and private images is not without tensions. 
Private images can never be neatly divided from public images; first, because any 
private image can always be republished in a public context; and second, because 
many of our camera practices, including our sense of ‘moment’ as it relates to photo- 
graphing the routines and rituals of everyday life, are learnt and internalized from 
the public image world. Private images always refract social mores. We know, for 
instance, that at least since the 1950s, parents have been the most fervent photog- 
raphers, and it is impossible to fully demarcate ‘natural emotion’ from the effect of 
a long history of marketing campaigns in the desire to photograph one’s children. 
Bourdieu’s pioneering study argued that the family photograph was simultaneously 
an index of family unity and a tool to achieve that unity: ‘Photographic practice only 
exists and subsists for most of the time by virtue of its family function of reinforcing 
the integration of the family group by reasserting the sense it has both of itself and 
its unity’ (1990: 19). But the family photograph arguably assumes its full—fetish- 
istic—force only when it represents a family unit that has all but disappeared. As 
Sontag observed barely a generation later: ‘A family’s photograph album is generally 
about the extended family—and often is all that remains of it’ (1977: 9). 

These remarks offer a useful frame for beginning to address the new image 
archives accumulating on social network sites such as Facebook and photo-shar- 
ing sites such Flickr. Many of the images uploaded to these sites—especially on 
Facebook—correspond to what Berger describes as images belonging to private 
experience.” In other words, the photographs were uploaded for viewing by (rela- 
tively) limited circles of family and ‘friends’. This distinctive mode of circulation 
runs counter to most modern forms of commercial publishing which sought broad 
and undefined audiences (even if they didn’t always find them). The capacity to 
give private photographs limited circulation within defined networks explains the 


level of trust many users invest in social network and photo-sharing platforms. In 
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such a setting, people may well circulate images showing unflattering, risky, dubi- 
ous or even illegal behaviour, in part to symbolize and enact the trust that they 
collectively share as ‘friends’. But, of course, ‘friends’ is an intensely ambiguous 
term in this context, and one that demands further consideration. While many users 
trust their photographs to social network platforms, it is the (always slippery) dif- 
ference between private and public contexts and uses that situates the widespread 
and intense concern expressed over the loss of control of ‘their’ photographs when 
they are uploaded to a site such as Facebook. This is not a matter of Facebook claim- 
ing ‘ownership’ over user images, as is frequently asserted. (It doesn’t, although it 
does arrogate certain rights to itself.) Rather, it concerns the inability of users to 
exercise precisely graduated control over the image, or to understand the implica- 
tions of failing to do so. Putting aside cases when images are accessed improperly, a 
common concern is the inability to set effective controls on the actions of others. For 
example, sharing photographs with ‘friends’ can result in them being made avail- 
able to ‘friends of friends’ with less restrictions than were mandated by the original 
uploader. And, once uploaded, it has been shown to be near impossible to ensure 
that unwanted images are actually deleted from websites such as Facebook (Cheng 
2012). (This touches the broader issue of data portability in relation to privately- 
owned, commercial websites in general.) 

I will give one example drawn from a small-scale ethnographic project examin- 
ing the way users employ Facebook to negotiate personal identity (Lambert 2011). 
Interviews revealed that a number of Facebook users had experienced situations 
where they felt they were cast as ‘voyeurs’ into a near stranger’s intimate life. 
This was an indirect function of the breadth and diversity of the ‘friends’ they had 
accrued over time, resulting in them being made privy to streams of personal pho- 
tographs from people who were casual acquaintances. Seeing such personal photo- 
graphs—ones that they felt they shouldn’t see—was described in terms that mixed 
discomfort and fascination. If this experience has some echoes of the commercial 
exchange in which a ‘celebrity’ sells the image rights to an intimate aspect of their 
‘private life’ (such as the birth of a child), it differs in its peculiar combination of 
selectivity (on Facebook, even as a relative stranger, you are part of a distinct net- 
work) and anonymity (who are these people?). More disturbingly for some users, it 
also raises the spectre of others looking into their personal photographs in this way. 

This example should not be read simply as an argument against using platforms 
such as Facebook to circulate personal images. Rather, it is to underline the extent 
to which the social practices accruing around photo-sharing sites still remain a 


‘black box’. There is an urgent need for empirical research regarding the sort of 
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photographs people actually choose to post and circulate, and for understanding 
the different meanings such images have for different viewers. There is also need 
for detailed analysis of the emergence of new lexicons employing online annotation, 
tags and other element to provide narrative contexts for reading such images (see 
Button 2009). 

Concern over the way that Facebook’s rules constrain user agency, including the 
capacity to exercise effective control over their own photographs or images in which 
they appear, has aroused some public controversy. However, the growing operation- 
ality of the image archive, and its availability, not to other users, but to the plat- 
form owner, has aroused less concern from users. The placement of large numbers 
of personal photographs onto privately owned platforms means that a whole reser- 
voir of once private life is now being opened up to corporate scrutiny. In the past, 
few would dream of showing their family photo albums to a company, or grant them 
permission to perform unspecified analyses on this corpus. Yet this is precisely what 
is happening today on a large scale. 

Is this a problem? What sort of analysis can digital images be subject to? Here 
we might distinguish common forms of data mining including registration and pro- 
file data, transactional data, user logs (which can provide analyses of social net- 
work ties) from the harvesting of metadata uploaded with photographs, and visual 
data from the photographs themselves. Image archives have always been search- 
able but, in the era of analogue photographs, this capability was limited by the 
physical logistics of classification and retrieval. Computerized archives of digital 
images sidestep these difficulties by enabling more flexible processes of categoriza- 
tion through metadata and user-generated tags, and by providing new protocols for 
searching and retrieving images. Today, metadata and tags are being supplemented 
by sophisticated forms of pattern recognition software, including facial recognition. 
Historically, these sorts of techniques were first deployed in relation to various offi- 
cial image archives, such as police databases of fingerprints: in other words, in rela- 
tion to various classes of ‘abnormal’ subjects.?? Numerous current popular software 
applications for processing, storing and sharing images, including Google’s Picasa, 
Apple’s iPhoto, Sony’s Picture Motion Browser (PMB), Windows Live Photo Gallery 
and, of course, Facebook now include facial recognition systems. In the process, 
the capacity to analyse image qualities is normalized and extended into the general 
population. 

In his work on ‘media visualization’ of large-scale image sets, Manovich suggests 
that automatic digital image processing techniques, aggregating qualities such as 


average brightness and saturation, number and properties of shapes, number and 
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orientation of edges, colours, can be deployed in conjunction with commonly avail- 
able metadata such as date/time and place to perform tasks such as mapping stylis- 
tic changes in news photography over history.” While excited by such possibilities, 
he further argues there is a need to actively prise the field away from its current 
leaders, in which analysis is seen primarily through a marketing lens. Such an ambi- 
tion presages the need for greater understanding and public discussion of the pro- 
tocols of the operational image archive. Manovich’s vision is predicated on the new 


availability of data sources via various social media platforms. 


For the first time, we can follow imaginations, opinions, ideas, and 
feelings of hundreds of millions of people. We can see the images and 
the videos they create and comment on, monitor the conversations 
they are engaged in, read their blog posts and tweets, navigate their 
maps, listen to their track lists, and follow their trajectories in physical 
space. And we don’t need to ask their permission to do this, since they 
themselves encourage us to do by making all this data public. (2011a) 


Part of the problem I am pointing to is precisely the fact that no one—and especially 
not the platform owners—needs to ask permission, even in relation to images that 
users categorize with the highest ‘privacy’ settings. The much-discussed trade-off 
between access to the service in return for the provision of personal data raises a 
raft of social, political and ethical issues that will not be easily or quickly resolved. 
In the current settings, capacity to search the new photo archives varies signifi- 
cantly, depending on who is doing the searching. For example, anyone can search 
Flickr’s public photographs using various key words, or general categories such as 
‘most popular’. Similarly, anyone can access public images on Facebook. However, 
even where a sizeable archive is publicly available (for instance the 5,668 photos on 
the 1,000,000 FACEBOOK ARTISTS page), they can only be displayed according to 
a standard grid format. This limits the capacity of independent researchers to treat 
the archive as ‘big data’. 

Some websites, including Flickr, have open Application Programming Interfaces 
(APIs) that allow access to certain image archives.** This has led to novel public 
research projects such as ‘Mapping the world’s photos’ which employed a dataset 
of about 35 million images collected from Flickr to plot the ‘spatial distribution of 
where people take photos’ and ‘to define a relational structure between the pho- 
tos that are taken at popular places’ (Crandall et al. 2009). Manovich (2011) argues 
these new modes of analysis based on automated pattern recognition techniques can 


be productively used to ‘defamiliarize’ the media environment around us. However, 
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the political implications of these undertakings remain ambiguous. Much data 
remains hidden behind private corporate walls. In this sense at least, the ‘torrent’ 
suggested by raw numbers can be misleading. The estimated 140 billion images held 
by Facebook do not constitute an archive in the sense that anyone can access that 
entire data set. Rather, that total is comprised by a multitude of micro-archives to 
which no one has complete access. The exception to this rule is, of course, Facebook 
itself. It is only from the optic of the platform host that ‘whole of population’ analysis 
becomes possible. 

Part of the problem is that something like Facebook’s photographic archive has 
been created in a context where few can really claim to understand how it is being 
used in the present, let alone how it might be used in the future. As Mark Andrejevic 


wrote in a cogent online post: 


There is an asymmetry to the so-called ‘end of privacy’: users are 
subjected to it, whereas those who control the commercial platforms 
are exempted in significant ways. ... We don’t have a clear idea of the 
experiments being conducted on us, the range of data collected about 
us, or how this data is used because these practices are not open and 
available to us. (2011) 


Andrejevic adds: 


There are huge emerging asymmetries in the terabyte world of ‘super 
crunching’: those with access to the databases and the tools for man- 
aging them can use data in quite different ways than those without 
access. (2011) 


This highlights the fact that it is not simply possession of data but possession com- 
bined with capacity to process data that is the key to power and prosperity in the 
digital milieu. For marketers, the visual archive harvested through social media is a 
new frontier, wide open for new analytic experiments. Imagine a scenario in which 
you have uploaded photographs to a social network website that incidentally reveal 
your private living space, including the clothes you wear, your furniture and appli- 
ances, the art works on your wall, and so on. Such image attributes could easily 
become analysable data, used in profile building and on-sold to third party advertis- 
ers. The worn fabric of your couch could become the basis for receiving a custom- 
ised ad about a special deal available on a similar item nearby. While some might 
welcome such targeting, the full implications of an unrestrained ‘knowing capital- 
ism’ (Thrift 2005) are disturbing. 
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Control of data is fundamental, not only to the business strategies of web 2.0 
companies, but to all companies in the ‘web squared’ world (O’Reilly and Battelle 
2009). The ability to access and manipulate data has become increasingly integral to 
the exercise of contemporary power (Lash 2010). The scale and depth of Facebook’s 
growing image archive suggests there is an urgent need to develop protocols for 
corporate ‘stewardship’ of such data collections that go well beyond current concep- 
tions of protecting individual privacy. Even without fully accepting Latour’s (2007: 
10) claims about the ability to read the individual ‘imagination’ from data traces, 
the corporate control of this enormous reservoir of data—‘freely’ contributed to by 
hundreds of millions of individual users—raises issues whose significance and scope 
might one day be compared to corporate ownership of the human genome. 

In concluding, I want to finally note the way that digital image archives are now 
popularizing certain standardized settings for organizing the ‘torrent’ of images. 
Common forms of metadata captured with digital photographs include date, time, 
place, camera type, and even camera settings such as aperture and exposure time.?° 
These values can all become grids for delimiting distinct sets of images. Two of the 
most popular modes of organization are geotagging and name-tagging (naming the 
people shown in the image). Increasingly, these operations can be automated via 
pattern and face recognition software. Services such as Google’s Panoramio and 
Microsoft’s Photosynth enable geotagged images uploaded to the web to be auto- 
matically integrated into ‘panoramic’ vistas of particular sites. Along another axis, 
Facebook (2011) recently automated the process for tagging ‘friends’ using face rec- 
ognition software. 

Automation of tagging functions may well prove popular, and, as with so much 
other technology, is inevitably presented as time-saving and efficient. However, as 
Thrift (2005) notes, automated technological settings all too easily disappear from 
consciousness. In some respects, the appearance of automated systems for organiz- 
ing photographs may be understood as a response to the problem of excess; they 
offer a convenient way of dealing with the torrent of images. But such a shift to 
automation also involves arrogating a portion of the difficult decisions about pho- 
tographic meaning to software. Making the world’s geographical co-ordinates a 
default setting for organizing images offers another twist in the endlessly fraught 
relation between photography and the physical world. Where the photograph was 
initially thought to guarantee the real as its objective image, and later was felt to 
be part of the world’s de-realization according to Baudrillard’s rather Borgesian for- 
mulation in which the map overtakes the territory, it is once again the world that 


now apparently functions to anchor the blizzard of photographs. Or, rather it is the 
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abstraction of place into digital geographical co-ordinates that becomes the key 
frame for locating photographs. The old problem of ‘reference’—of how the image 
relates to the world it re-presents—is displaced onto the terrain of metadata and 
algorithmic analysis. 

Automated name-tagging using facial recognition software is currently gener- 
ating some disquiet among users, presumably because of its proximity to official 
techniques of biometric surveillance. It remains to be seen to what extent the ‘con- 
venience’ of this new colonization of intimate life will be accepted and internalized. 
But the mere fact of the expansion of biometric techniques into archives of ostensi- 
bly personal images indicates the extent to which the ‘operational archive’ of digital 
images no longer belongs to the disciplinary institutions in which it had its origins 
in the nineteenth century. Rather, it is symptomatic of the new user-generated mode 
of governmentality that Lash (2010) calls ‘intensive culture’, in which the exercise of 
power is mobilized through data that is not gathered from above, but is largely self- 
generated and self-organized in an ongoing process. Like the routine of photography 
in the age of always-on, always-available mobile phones, data gathering no longer 
depends on the scrutiny of ‘special’, ostensibly abnormal subjects or activities, but 


has become co-extensive with ‘life’. 
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Notes 


1. Osborne adds the caveat: ‘but without the iconic aspect of perceptual resemblance 
previously associated with them’ (2010: 64). 


2. Osborne further argues that the emergence of ontological anxiety surrounding the 
photograph is distinct to the deployment of digital images, but is rooted elsewhere—in 
social relations of abstract exchange. 


3. I would immediately add that any distinction between ‘professional’ and ‘everyday’ 
images is not hard and fast, and the digital threshold is, in part, defined by increasing 
porosity between the two. However, as will become clear below, my primary concern 
in this essay is with ‘private’ rather than ‘public’ images. 


4. Flickr was launched by Ludicorp in 2004 and acquired by Internet giant Yahoo one 
year later. 


5. See http://www.flickr.com/cameras, accessed 6 December, 2013. The figures are only 
accurate to the extent that Flickr can automatically detect the camera used, which it 
reports as about two-thirds of the time. 


6. http://blog.flickr.net/en/2011/08/04/6000000000/ 


See Library of Congress 2010: x. This number is exclusive of moving images, posters, 
prints and drawings. 


8. The May 2009 figure is at http://www.facebook.com/blog.php?post=87157517130 
The February 2010 figure is http://blog.facebook.com/blog.php?post=206178097130 


9. The figure is given in a blog post by Justin Shaffer (26 August, 2011) available at http:// 
blog.facebook.com/blog.php?post=10150262684247131 


10. In response to the question ‘How many photos are uploaded to Facebook each day?’, 
Mitchell, who identifies himself as a Facebook engineer, writes: ‘We currently have 
over 200 million photos uploaded per day, or around 6 billion per month. There 
are currently almost 90 billion photos total on Facebook. This means we are, by 
far, the largest photos site on the Internet’ (response dated 26 January, 2011). 
http://www.quora.com/How-many-photos-are-uploaded-to-Facebook-each-day/ 
all comments/Justin-Mitchell 


11. Sontag writes: ‘A photograph is only a fragment, and with the passage of time its 
moorings come unstuck. It drifts away into a soft abstract pastness, open to any kind 
of reading’ (1977: 71). 


12. Barthes argued: ‘It is not the image which comes to elucidate or “realize” the text, but 
the latter which comes to sublimate, patheticize, or rationalize the image’ (1983: 204). 


13. Elsewhere I have argued that photographic culture played a major, if often 
unacknowledged, role in establishing the conditions in which semiotic openness and 
textual polysemy move from the margins to the mainstream of contemporary thought 
(McQuire 1998). 


14. Manovich offers the standard computer science definition of ‘big data’ as ‘data sets 
whose size is beyond the ability of commonly used software tools to capture, manage 
and process ... within a tolerable elapsed time’ (2011a). 
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15. 


16. 
17. 


18. 


19. 


20. 


21. 


22. 


23. 


Latour (2007) argues that the entire divide between social and psychological modes of 
research is becoming less relevant. Similarly, Savage and Burrows suggest: ‘Our point 
was rather that these familiar debates—quantitative versus qualitative—are now 
much less salient in a world of digitized information with much more interplay between 
numbers, narratives and images’ (2009: 765). For Wired (2008), the equation is far 
simpler, amounting to the end of theory itself: ‘This is a world where massive amounts 
of data and applied mathematics replace every other tool that might be brought to 
bear. Out with every theory of human behavior, from linguistics to sociology. Forget 
taxonomy, ontology, and psychology. Who knows why people do what they do? The 
point is they do it, and we can track and measure it with unprecedented fidelity. With 
enough data, the numbers speak for themselves.’ 


Benjamin is quoting Danish writer Johannes Jensen (1999: 276). 


Sekula argues: ‘These two semantic paths are so fundamental to the culture of 
photographic realism that their very existence is usually ignored’ (1986: 18). 


When Charles Darwin used seven sets of photographs in his The Expression of the 
Emotions in Man and Animals (1872: vi), he argued they ‘are much superior for my 
purpose to any drawing, however carefully executed’. However, the excessive 
particularity of photo-realism, in comparison to more traditional engravings 
whose function was to synthesize a multiplicity of possible examples into an ‘ideal’ 
representation, aroused considerable debate in the scientific community. 


One of one Galton’s images formed the frontispiece of Havelock Ellis’ The Criminal 
(Sekula 1986: 40-46) 


Digital photography revives the old ‘generalist’ trajectory in distinct ways. The cover 
illustration of Time (1993), headlined ‘The new face of America’, shows a woman’s 
face ‘created by computer from a mix of several races’. Inside we read that this very 
Kuleshovian ‘new woman’ is a composite creature created through digital ‘morphing’ 
of specified amounts of ethnicity: 15% Anglo-Saxon, 17.5% Middle Eastern, 17.5% 
African, 7.5% Asian, 35% Southern European, and 7.5% Hispanic. See http://www. 
time.com/time/covers/0,16641,19931118,00.html. Today, rather than merely updating 
Galton’s procedure with reference to demographic statistics, such an image could 
potentially be created by direct analysis of a large scale image archive. 


Both Flickr and Facebook offer variable settings in terms of ‘public’ and ‘private’ 
display. As a dedicated photo-sharing website, Flickr is heavily patronised by photo- 
enthusiasts and professional photographers who make many of their images publicly 
accessible. Conversely, the bulk of Facebook’s user-generated image archive is not 
accessible to the public. 


Automatic Fingerprint Identification System (AFIS) was introduced from the late 
1970s. Facial recognition systems were initiated in the 1960s, but did not enter 
widespread until the 1990s, surfing the same socio-technological wave that drove the 
upsurge of personal computers and digital cameras. 


Manovich distinguishes media visualization from data visualization in terms of process 
and output: ‘Typical information visualization involves first translating the world into 
numbers and then visualizing relations between these numbers. In contrast, media 
visualization involves translating a set of images into a new image which can reveal 


24. 
25. 
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patterns in the set. In short, pictures are translated into pictures’ (2011b). However, it 
should be acknowledge that the security of this distinction is compromised in an era 
in which (digital) pictures are fundamentally comprised of numbers. 


See http://www. flickr.com/services/apps/about 


As O’Reilly and Battelle comment: ‘Initially users taught their computers the 
association between photos and locations by tagging them. When cameras know 
where they are, every photo will be geotagged, with far greater precision than the 


humans are likely to provide’ (2009: 5). 
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CHAPTER 7 


Lights, Camera, Algorithm: 
Digital Photography’s Algorithmic Conditions 


Daniel Palmer 


Even as its symbolic apparitions are endlessly animated across the cultural field, 
photography as we once knew it is all but over. Cameras have become computers, 
and mobile phones have become cameras, and in the process the idea and practice of 
photography has been thoroughly, if not completely, transformed. Computerisation 
entails a fundamental and well understood shift that barely needs repeating—from 
photography as the transcription of light onto a light sensitive material surface to 
photography as the translation of light into digital data. Camera phones introduce 
new levels of ubiquity and immediacy. But the implications of these developments 
are continually unfolding. For obvious reasons, critical attention has turned from 
a focus on the importance of manipulability and photography’s supposed truth sta- 
tus that preoccupied writers on digital photography in the 1990s (Mitchell 1992), 
to questions about photography’s practices and circulation, particularly on the 
Internet (Hand 2012; Lister 2013). Increasing attention is now also being directed 
towards the role of algorithms, those step-by-step procedures for calculations which 
lie at the core of all data processing. Thus William Uricchio (2011) has recently 
described imaging software such as Photosynth as part of an ‘algorithmic turn’, spe- 
cifically concerned to understand new computationally enhanced viewing positions 


enabled by the aggregation of location-tagged photographs, while Daniel Rubinstein 
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and Katrina Sluis argue that ‘the image has to be considered as a kind of program, 
a process expressed as, with, in or through software’ (2013: 29, original emphasis). 
In this chapter, I wish to explore what I am calling photography’s algorithmic condi- 


tions, that is, the image’s near total dependence or conditionality on the algorithmic. 


Compositional Conditions: From the Decisive Moment 
to the Expanded Frame 


For the famed French photojournalist Henri Cartier-Bresson, the narrow rectangle 
of the Leica 35mm viewfinder opened up a new world and a new kind of seeing—one 
that was spontaneous and unpredictable. Thus in Images a la sauvette, his influen- 
tial 1952 book of photographs, he elevated ‘snap shooting’ to the level of a refined 
and disciplined art. At its core was the notion of the ‘decisive moment’, in which all 
the formal compositional elements of the picture come together at the right moment. 
The photographer, he wrote, ‘composes a picture in very nearly the same amount 
of time it takes to click the shutter, at the speed of a reflex action’ (Cartier-Bresson 
1999: 33). Thus, in this account, the aesthetic faculties of modern photographers 
have accelerated to match the speed of their apparatus amid the bustling world of 
the twentieth century. Cartier-Bresson summarized his aesthetic approach: ‘to take 
photographs means to recognize—simultaneously and within a fraction of a sec- 
ond—both the fact itself and the rigorous organization of visually perceived forms 
that give it meaning’ (1999: 16). In a sense, Cartier-Bresson was engaging in a form 
of what we might now call pattern recognition. 

Cartier-Bresson’s style of photography is still possible, still practiced and cele- 
brated, but its importance is marginal. With the digital universe, other types of pho- 
tography have become more culturally significant, ones which often involve a shift 
from the single moment of capture to the expanded moments of post-production. 
We can see this most dramatically in the work of artists, particularly in the work of 
Andreas Gursky and Jeff Wall, two figures who have perhaps most successfully capi- 
talised on the visual potential of the digital photograph. Both Gursky and Wall were 
pioneers of the use of digital imaging at the start of the 1990s. The German artist, 
Gursky, started to work digitally after finding traditional documentary photogra- 
phy was ‘no longer credible’ (Gursky, quoted in Ohlin 2002: 29), and is now closely 
associated with a form of heightened, or hyper, realism which emphasises formal 
elements and is often interpreted as making visible the invisible flows of global capi- 
tal. Wall, a Canadian artist, is more invested in the history of picture making in 


art history, and his use of digital imaging is more subtle. Consider his monumental 
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Figure 1. Oscar Rejlander, The Two Ways of Life, 1857, combination print from multiple negatives. 


work A Sudden Gust of Wind (after Hokusai) (1993). A transparency in light box 
that stands at nearly four metres wide (250 x 397), it is one of Wall’s most ambi- 
tious and complex ‘cinematographic’ works, making a direct allusion to art history, 
remaking a colour woodcut by Japanese artist Katsushika Hokusai in contemporary 
Vancouver. Perhaps most importantly, Sudden Gust is a photograph that juxtaposes 
movement and stasis. In any of the many texts you might read about this work, one 
will learn that it was constructed from at least fifty separate images—one hundred, 
according to some critics—shot from the same single camera position, taken over 
the course of one year. Indeed, Wall told me himself that the reason it took a year is 
that he had forgotten to photograph the leaves on the ground, so—in a curious con- 
cession to nature—he had to return a year later. Finally, the multiple photographic 
elements have been assembled in the computer, and digitally montaged, to produce 
a seamless effect. 

There are, of course, countless precedents for the montage process, going back 
right to the earliest uses of photography. For his elaborate allegorical work The 
Two Ways of Life from 1857, Oscar Rejlander combined at least thirty negatives to 
produce one of the best-known and most infamous combination prints. The various 
pieces were carefully joined, and then rephotographed to create the illusion of a 
single scene. Even in the field of documentary, manipulation has a long history long 
before Photoshop—as any student of the history of photography knows. For instance, 
Mia Fineman (2013) tells the story of how, following the end of the American Civil 
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War, photographer Matthew Brady made a group portrait, Sherman and His Generals 
(1865). Brady asked Sherman and his chiefs of staff if he could make a group portrait, 
but unfortunately one of the generals was late. When everyone, exhausted from the 
battles, needed to go home to their families, Brady took a picture and later, when the 
tardy general showed up, he photographed him alone before carefully combining the 
two negatives in the darkroom to produce the single group picture. Interestingly, as 
Fineman points out, when Nancy Pelosi organised a group portrait of the Democratic 
Women of Congress in January 2013, and an identical solution was used to overcome 
a similarly late sitter, it generated considerable controversy. However, at one level, 
both Brady’s image and the image of the Women of Congress could be said to be 
accurate representations, in that they do ‘not represent a single moment in time, but 
a single moment in history’ (Fineman 2013). 

Wall’s work is not concerned with photographic truth. However, Sudden Gust of 
Wind is a particularly arresting image in terms of the aesthetics of photographic 
composition, because it conspicuously relies on photography’s ability to capture a 
fleeting moment. By deploying the idea of the papers flying in the wind, the picture 
‘clearly place[s] itself in the tradition of the aesthetic of the instant’, as Laura Mulvey 
(2007: 30) has astutely observed, and thus also of the spontaneity and realism of the 
‘decisive moment’ school of photography embodied by Cartier-Bresson. It has even 
been suggested that Sudden Gust ‘could be taken as a joke at the expense of snap- 
shot photography’ since ‘here spontaneity is not caught by the photographer-hunter, 
but constructed using the most patient and unspontaneous means’ (Newman 2007: 
148). To be sure, as in many of Wall’s works, the goal appears to be that of creating 
a tension in the image, distilling a moment of action. However, Sudden Gust breaks 
with the conventional idea of a photographic temporality, the capital ‘T’ Time that 
Barthes invests in, as a piece of a past moment that exists in the present. Instead, 
the image is now composed of multiple moments. The final image is a ‘synthetic 
combination of multiple indexes’ (Van Gelder and Westgeest 2011: 38). It departs 
fundamentally from ‘the tiny spark of contingency, of the Here and Now, with which 
reality has seared the subject’, as Walter Benjamin famously described photogra- 
phy ([1931] 1979: 243). Instead, through seamless pictorial unification, Wall creates 
what he has called a ‘complex illusion of instantaneousness’ (Wall 1996: 87). In so 
doing, as Mulvey puts it, the work ‘inscribes the temporality of pre-photography on 
to the temporality of post-photography’, wherein ‘[jJust as Hokusai could only aspire 
to the instantaneity of photography, Wall commemorates its loss’ (2007: 37). 

The treatment of the wind in Wall’s photograph is particularly suggestive. Wind 


is a constant presence in photography, even where it is not identified as its subject. 
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Figure 2. Patrick Pound, Portrait of the Wind, 2010 (detail). Courtesy of the artist 
and Stills Gallery. 


The Australian artist Patrick Pound has collected an entire series of found photo- 
graphs that involve the wind, to produce a collage work called Portrait of the Wind 
(2011), featuring women with billowing dresses and men with tousled hair and way- 
ward ties. Of course, the wind is not itself visible in these images, only its effects. 
Indeed, in Wall’s work, given the image is digitally montaged, it is in fact possible 
that the wind may have never have existed at all, and that we are witnessing effects 
without a ‘natural’ cause. Sure enough, the ‘sudden gust’ was produced by a wind 
machine, and the hat was tied to a stick with nylon thread. As Wall says: 


The montage is composed of acts of photography, even if there is no 
simple photographed moment. I don’t think any photographic qualities 
are eliminated, except the single moment in which the entire image 
was made. I admit that may be the decisive absence, but I like to make 
a picture that derives from that absence and contemplates it. (Quoted 
in Tumlir 2001: 116) 


Here, the decisive moment has thus become the decisive absence, precisely as pho- 
tographic space has expanded and multiplied indefinitely. This is now the condition 
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under which all photographs are poten- 
tially produced. I am reminded of this 
fact whenever I take a photograph with 
my iPhone using the High Dynamic 
Range (HDR) setting, which automati- 
cally blends together three differently 
exposed versions of the ‘same’ scene in 
an effort to improve details in the shad- 
ows and highlights. Since the world 
does not usually stand still to be photo- 
graphed, movement in the frame leads 
to unusual repetitions, ghostings and 


dismemberments. Even stranger results 
were generated in the glitchy Apple Maps Figure 3. An example of ‘ghosting’ using the iPhone 4S’s HDR 
. wea exposure setting. Photo: Daniel Palmer. 

app in 2012, when cities were peppered 
with warped buildings and distorted 
bridges in the effort to render 3D geometries from 2D images. Galleries of these mis- 
fit renders can now be found on the Internet, as an ode to ‘algorithmically-authored 


oddities’ (Vanhmert 2013). 


Camera Conditions: From Apertures to Augmented Reality 


Over forty years ago, in a famous work of photo-conceptualism, the British artist 
John Hilliard laid out what he understood to be the fundamental procedures of pho- 
tography. The work, Camera Recording Its Own Condition (7 Apertures, 10 Speeds, 
2 Mirrors) (1971) comprises a gridded display of seventy photographs, arrayed in 
ten rows of seven across, taken by a camera aimed at a mirror, showing itself at the 
moment of exposure. As the second part of the title of the work indicates, the snap- 
shot-style images, which move from pure white to pure black, are the result of all 
the possible combinations of aperture size and shutter speed in Hilliard’s camera, 
an East German made Praktica. In the grid, the artist has positioned the optimal 
‘correct’ exposures in a diagonal line from the top right to bottom left. This chang- 
ing of the mechanics of each shot reveals the intention of the unseen photographer. 
Importantly, Hilliard’s fingers can be seen operating the camera. Hilliard also holds 
up a smaller mirror which reflects and makes legible the camera’s setting and con- 
trols, making it a self-portrait of sorts. 
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Figure 4. John Hilliard, Camera Recording Its Own Condition (7 


notes, ‘Photography is both the medium 
and the subject of the work, giving not 
a picture of “reality”, but different ver- 
sions of reality’ (Tate). In addition to 


Apertures, 10 Speeds, 2 Mirrors), 1971. Courtesy of the artist questioning photographic objectivity, as 


and the Tate. 


a work of conceptual art, it is likely, as 

photographer Jeff Wall has argued, that 
Hilliard’s work is also a critical commentary on what many artists at the time saw as 
‘an excessive subjectivism in the then-reigning versions of art photography’ (2012: 
698). This is the famous deadpan (non-)aesthetic of conceptual photography, particu- 
larly that which operated in serial mode, in which the individual creative agency of 
the photographer is purposely downplayed: here, the camera is simply ‘recording’, 
as if it has an agency of its own (Palmer 2013). 

Viewing Hilliard’s work today, however, with the knowledge of the increasingly 
sophisticated automation of cameras that was about to occur, starting with basic 
electronic controls in the 1970s and massively expanded with digital cameras since 
the 1990s, we can now interpret this piece afresh. Arguably, what the work becomes 
now, above all, is a homage to the economic simplicity of the camera’s ‘condition’ 
in the classic age of 35mm analogue photography. Today, the activity of taking 
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photographs—of using a camera—is at once both simpler and more complex than 
the conditions depicted in Hilliard’s work. Taking a correctly exposed and focused 
photograph is certainly simpler, to the extent that most cameras are now completely 
automated. Most people who use a camera today simply do not concern themselves 
with aperture, shutter speed or focus. To be sure, this has been the case since Kodak 
pioneered its brilliant logic of ‘you press the button we do the rest’ at the end of the 
nineteenth century. However, automatic here effectively meant fixed (aperture and 
shutter speeds)—Box Brownie cameras could not automatically respond to changing 
conditions. Moreover, the ‘we’ in Kodak’s statement has now shifted from a company 
producing prints for a consumer to the algorithmic software in which photography is 
now embedded (from camera design to online photosharing tools). 

The design of digital cameras is superficially similar to traditional cameras. 
Some of the available options—such as auto-focus—were pioneered in electronic 
cameras from the 1970s. However, a film camera could never interpret a visual 
scene with anything that might be called intelligence. In a digital camera, given 
that the data collected by the sensor array in response to a pattern of illumination is 
essentially a string of digits, the level of intelligence is limited only by the software, 
that is, the algorithmic processes that operate on the raw data. Thus auto-white bal- 
ance, face-recognition and enhancing the sharpness of an image became standard 
features on digital cameras almost immediately, not to mention the default compres- 
sion protocol of the JPEG and its universal standardisation of metadata. But the pos- 
sibilities of intelligent operations are expanding in range and effect, given that new 
algorithms are constantly being developed. As cameras are increasingly networked 
to the Internet, software is now available that can ‘learn’ to fix up pixelated pho- 
tos, and even ‘complete’ scenes in photographs, through the analysis of millions of 
related images in a database. Google’s social media site has implemented sophisti- 
cated facial recognition so it can more finely auto-tune images to fix a person’s blem- 
ishes or skin tones. As with so much algorithmic calculation, all of these techniques 
rely on laws of averaging (for example, to fix noise, the software measures the statis- 
tical properties of images and creates an algorithm that determines what is or is not 
‘noise’ in any given photograph). 

For a camera user, these actions can happen more or less seamlessly. But the 
processes by which these actions happen are usually opaque. Vilém Flusser (2000) 
famously wrote of the camera as the archetypal black box, or more specifically as a 
programmable apparatus that paradoxically programs the photographers (function- 
aries) who use it. However, Flusser’s critique is more complex than often recognized, 


counterbalanced as it is by his praise for what he calls ‘experimental photography’: 
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‘to create a space for human intention in a world dominated by apparatuses’ (2000: 
75). Ironically, his optimism in this regard now reads as a relic of the analogue era. 
For as the photograph becomes a data object, it also becomes part of networked 
visual culture online, such that the camera’s ‘condition’ is no longer separable from 
the networked image culture in which it is enmeshed. This helps to explain why cam- 
era manufacturers now increasingly market their products not only on the quality of 
their hardware (such as lenses, traditionally the benchmark of quality, now increas- 
ingly interchangeable), but on the quality or features of their software (algorithms). 
I have written elsewhere of the marketing for the Olympus OM-D, released in 2012, 
which presents itself as a camera that ‘aims to change the way in which you experi- 
ence photography’ specifically on the basis of innovative software that enables cre- 
ative intervention at the point of capture (Palmer 2013). Needless to say, such algo- 
rithms involve complex patents and are closely guarded commercial secrets. 

For the everyday user of a camera—particularly those embedded in mobile 
phones, which have quickly become the majority—the nature of the labour involved 
in taking a photograph is dramatically changing. In her famous moral critique in 
the 1970s, Susan Sontag argued that ‘through the camera people become custom- 
ers or tourists of reality’ and that photography’s ‘main effect is to convert the world 
into a department store or museum-without-walls in which every subject is depreci- 
ated into an article of consumption, promoted into an item for aesthetic apprecia- 
tion’ (1977: 110). This critique still holds, but has intensified. Today, aesthetic appre- 
ciation is not even required for the world photographed to become a consumption 
point. On the one hand, photography today has become increasingly conversational 
and transactional (as in all the ephemeral photo-messages sent by phone). But on 
the other hand, photography has become more instrumental, and the act of pho- 
tographing has increasingly become a consumer event—whether because inserted 
into an online entertainment activity (Common on mobile phones), or as a vehicle 
for augmented reality that in its dominant forms (QR codes) is better described as 
‘augmented advertising’. When geo-location is added to the equation, urban space 
has become available for commodification and consumption in new ways, in the 
guise of photographic activity. For instance, Nokia City Lens, a typical piece of 
augmented reality software, ‘gives dynamic information about users’ surroundings 
such as shops, restaurants, and points of interest, shown as virtual signs overlaid on 
or above buildings.’ Apps like these are challenging the very definition of photog- 
raphy itself. 

Meanwhile, for the erstwhile creative photographer, as we have already seen 


above, decision making can increasingly be left until after the moment of exposure 
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(Palmer 2013). When, after all, is a photograph ‘made’? As Geoffrey Batchen has 


observed, the answer to this question has always been complex: 


Is it when the photographer depresses the camera shutter, submitting 
a chosen scene to the stasis of framed exposure? Is it when the pho- 
tographer singles out this exposure for printing, thereby investing a 
latent image with the personal significance of selection, labour, and 
most crucial of all, visibility? Or it when that image is first exposed to 
the public gaze. (2001: 83) 


In light of this, we can recognise that the emphasis on pre-visualisation, the imagi- 
native period before the moment of exposure, associated most closely with Ansel 
Adams, is not that different from Cartier-Bresson’s ‘decisive moment’ with its 
emphasis on the fraction of a second involved in choreographing a photograph. In 
reality, both of these approaches involve a certain myth-making around the creative 
visionary engaged in a poetic encounter with the world. Each downplays the mediat- 
ing process, including the crucial activity of editing, preferring to emphasis an artis- 
tic method of subjective intuition. At the opposite extreme to Adams and Cartier- 
Bresson is the photographer Garry Winogrand, who famously claimed to photograph 
to see what the world looks like in photographs, and eventually turned to a motor- 
winder to speed up the process of capture, ensuring that editing became the critical 
part of his creative process. Notably, the topic of editing has been largely overlooked 
in photographic history and theory (Campany 2013). 

With digital tools, the deferral of creative decision making can take many unex- 
pected directions. Photographers have long lived with the possibility that nega- 
tives may be able to be printed in new (not necessarily better) ways in the future. 
It has often been possible to print higher quality versions of images than when they 
were original taken (or in the photographer’s own lifetime), thanks to technologi- 
cal advances in lenses and paper quality. Today, these potential improvements have 
been virtualised. Thus the most recent version of Photoshop can process RAW files 
better than the previous versions of Photoshop. That is, there is now always the 
promise that RAW sensor data might be processed better in the future, giving a 
new meaning to the conventional notion of the latent image. Furthermore, the image 
might not just be produced at a higher quality, but the data could be interpreted 
in completely new, as yet unrealised, ways. Thus the image operates ‘in a constant 
state of deferral’ of its potential (Rubinstein and Sluis 2013: 29). 

A very different recent example of the use of algorithms gives a taste of where 


editing might be heading in the age of digital automation. Google recently announced 
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the introduction of new photography features in Google+, in an effort to compete 
with Facebook and their takeover of Instagram. Google’s pitch is an attempt to make 
the human labour of editing unnecessary or obsolete. By using algorithms, a feature 
called Auto Highlight will skip over blurry photos, duplicates, and underexposed 
shots, and choose the ‘best’ photographs to display and share. The example is given 
of uploading hundreds of new vacation photographs: 


Google Plus [sic] will surface the best ones to share with others by 
eliminating blurry or unfocused photos and even analyzing whether 
people look happy. Using Knowledge Graph, it might recognize where a 
person is located and prioritize photos taken at important landmarks. 
It will also prioritize photos of people recognized to be in a user’s close 


Google Plus circles, like family members. (Leber 2013) 


In other words, drawing upon the computing power, machine learning, algo- 
rithms, semantics analysis and other innovations that have established Google’s 
search engine as the most influential force on the Internet, the company is effec- 
tively offering to edit your photographs for you. Taking ‘smile detection’ to another 
level, one feature promises to automatically stitch together a single image from sev- 
eral, for example from a sequence of photos of a group of people, so that everyone’s 
eyes are open and everyone is smiling. The innovation here lies at the level of auto- 
mation. All of this is designed to save people the time and trouble of choosing and 
editing photos, premised on the claim, as Google representative Vic Gundotra puts 
it, that: ‘if we are honest with each other photos are very labor intensive’ (quoted in 
Liedtke 2013). The plans even include ‘a photo rating system’ which is being trained 
by hundreds of human raters, designed to help the photo engine begin to account for 
‘aesthetics and human tastes’ (Gundotra, quoted in Leber 2013). 


Temporal Conditions: From Emanations of the Past to 
Calculations of the Future 


If we accept Google’s anticipation of consumer expectations, photography is no 
longer exclusively concerned with recording a past moment. Indeed, the tempo- 
ral question in photography can now be said to also include both the time involved 
in editing and sharing those moments. And that whole process must aspire to be 
as automated and instantaneous as speech itself at least superficially appears to 
be. One is reminded of a 2011 commercial for Apple iCloud in which a child rushes 
inside a house to view photographs on an iPad that have just been taken of him 
playing outside with an iPhone. They have seamlessly and remotely synchronised 
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themselves from the capture device to a 
different display screen. Moreover, they 
have been uploaded to the network and 
then downloaded again—which in prac- 


tice can entail some delay, not that you 


would know this from the Apple com- Error 
mercial, with its reassuring voiceover 
‘capture the moment here, and it’s wait- Time is over a day old. 


ing for you there’. The so-called ‘Photo 
Stream’ is indeed designed to auto- 
mate the sharing of photographs not 


only between devices but also between 


family and friends (using Shared Photo 


Figure 5. Apple TV Error Message, 2013. Photo: Daniel Palmer. 


Streams). The seamless nature of the 
process evokes a fantasy of instanta- 
neous distribution and ubiquitous photographic presence. But even Apple’s digital 
universe is not without its asynchronous moments. Recently, during an attempted 
viewing, an error message appeared on my display screen which read: ‘Error. Time 
is over a day old.’ This nonsensical yet strangely poetic phrase, generated without 
warning or further explanation by the software of my Apple TV unit, raises the ques- 
tion: can time be old? In computerisation, where now is the only time that matters, it 
certainly appears that time can indeed be too old. 

For many of its most important theorists, time is the very essence of photogra- 
phy. Not the now nearly instantaneous time between recording and displaying, but 
the very possibility of a time-travelling encounter between the past and the present. 
Thus in his classic formulation of the particular quality of (analogue) photography, 


Roland Barthes famously wrote: 


The photograph is literally an emanation of the referent. From a real 
body, which was there, proceed radiations which ultimately touch me, 
who am here; the duration of the transmission is insignificant; the pho- 
tograph of the missing being, as Sontag says, will touch me like the 
delayed rays of a star. (1984: 80-81) 


This passage is of course an enthusiastic defence of photography’s specific index- 
ical and temporal qualities. Barthes’ use of the word emanation is self-consciously 
Romantic, with its overtones of cosmology and certain religious systems (the word 


derives from the Latin emanare meaning ‘to flow from’ or ‘to pour forth or out of’, 
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as the mode by which all things are derived from the First Reality, or God). It is true 
that the referent, in this case the ‘real body’, has caused a particular reflection of 
light that has in turn caused a specific reaction on the part of chemical materials 
that made the photograph. However, Barthes makes a poetic leap to suggest that the 
photograph somehow relays radiations of the past body to the living body who views 
a photograph at a later date. Needless to say, what we actually see when we look at a 
photograph is something less than an emanation; traditionally, we see a mere piece 
of paper, a flattened, two-dimensional, often black-and-white image of the world that 
bears an impression of light according to the specific optical and chemical technolo- 
gies that brought that image into being. Today, we more likely view an arrangement 
of pixels on a screen. 

Nevertheless, Barthes’ seductive notion of the delayed rays of the star remains 
highly suggestive. Since the light from our closest stars (beyond the sun) takes over 
four years to reach the earth—and that of distant stars may take thousands or even 
millions—when we look at stars we are, in a sense, looking back in time. We are 
indeed looking at old light. While the finite speed of light has been known for centu- 
ries (photons travel at a constant speed of about 300,000 kilometres per second) we 
tend to assume its instantaneity, forgetting the gap between reality and its appear- 
ance—which in this case is large indeed. Translated to photography, this idea of 
time-travelling ‘radiations’ forms a crucial part of Barthes’ broader ontology of pho- 
tography, by which photographs offer a co-presence with the past, the famous ‘that- 
has-been’ tense of photography (Barthes 1984: 96). A photograph records the time 
at which reflected light strikes the film or plate, and is thus both an indexical trace 
of the object and a certified document of time itself. This is the ultimate, overarch- 
ing philosophy of photography in Camera Lucida: that the essence of photography is 
Time itself (Barthes 1984: 96). Indeed, in almost all classical accounts of photogra- 
phy, it is a retrospective medium, charged exclusively with looking back—with his- 
tory, memory and nostalgia. 

All of this remains true of many forms of digital photography, to the extent to 
which they continue to serve the aim of documenting moments of time. However, 
digital photography is also capable of functioning in very different ways. We have 
already seen that the ‘duration of the transmission’, contra Barthes, is now extremely 
significant. In its new calculative and distributed dimensions, photographic images 
can behave in much more dynamic and performative ways. By way of a dramatic 
example, consider a recent Australian smart phone app developed to prevent graffiti 
tagging, VandalTrak (www.vandaltrak.com.au). As we know, the native camera app 


on an iPhone, or any other smartphone, is just one piece of software among hundreds 
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that can engage the camera hardware. Apps are simply small pieces of software that 
add functionality to a digital device, and in the case of photo apps they tend to have 
two main purposes: first, to make images more ‘artistic’ or aesthetically distinc- 
tive (such as Hipstamatic); and second, to facilitate the distribution or publication 
of photographs (such as Snapchat). The most successful of all, Instagram, does both. 
VandalTrak belongs to a subset of the second category. Like many apps, it takes 
advantage of the distinctive feature of camera phones, such as the iPhone, to auto- 
matically tag photographs with their GPS location in the metadata, allowing images 
to be browsed and arranged geographically, and for software to aggregate them for 
various purpose. A number of photo-sharing websites, such as Flickr (www.flickr. 
com) and Panoramio (www.panoramio.com), now provide large collections of pub- 
licly available, accurately geo-tagged images. 

The VandalTrak app has been developed on the assumption that graffiti is a sig- 
nificant problem, ‘reducing the value and image of our communities and neighbour- 
hoods’, as the website claims. It works by using an online system that seeks to log, 
track and manage graffiti incidents in one place. Members of the public can take 
photographs of graffiti when they see it and upload it to the VandalTrak online data- 
base, where it is automatically logged by its location, catalogued by the traits of the 
tag (using pattern recognition software) and provided to police, council and other 
organisations (Olding 2013). Thus, with its citizen policing logic, VandalTrak is part 
of a wave of what has been called ‘participatory sensing’ enabled by mobile media, 
using embedded devices to capture data about oneself and one’s community (often 
extending earlier histories of citizen science projects). As Kate Shilton has argued, 
participatory sensing is an emerging form of mass data collection that may, under 
appropriate conditions, be conceived as a form of ‘empowering’ self-surveillance—an 
opportunity for individuals and groups to ‘provide possibilities for self-exploration, 
community discovery, and new knowledge creation’ (2010: 132). For instance, peo- 
ple can use a mobile phone’s sensor capacities, of which the camera is one, to work 
together for specific purposes, from such simple projects as mapping local pollution 
to targeting and identifying invasive plant species in national parks. VandalTrak is 
indeed a form of community intelligence-gathering, and has been embraced by the 
Australian police who encourage the public to anonymously provide information 
about the identity of taggers. 

The contrast between the photographs that Barthes write about in Camera 
Lucida and the digital images produced for VandalTrak could not be greater. The lat- 
ter operates in a purely instrumental manner, existing only to be communicated as 


evidence, and put to work in an economy of power (Tagg 1988). Many writers have 
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suggested that the move to digital involves a shift from thinking about photography 
exclusively as representation and increasingly as information. Perhaps it is better 
to think of this in terms of a bias, given that photographs have long been used as 
information within institutional archives. But with VandalTrak the significance of 
the new bias is clear. What matters is not the individual viewer of the photographs, 
as in Barthes’ phenomenological method, but that digital light—or the electrical sig- 
nals that produce the picture elements (pixels) that we see on the screen—are here 
distributed across databases and interpreted by computer algorithms. Rather than 
the delayed rays of a star, the more accurate comparison is the real-time screen of 
radar, which as Lev Manovich observes is the precursor to today’s real-time screen 
image (2001: 99). These real-time screens are capable of instant changes in out- 
put, generating dynamic images experienced without delay and typically driven by 
some form of user interactivity. They alternate between windows and control panels, 
between the dimensions of ‘representation’ and ‘control’, in which the subject oscil- 
lates between the roles of viewer and user—inviting us to be both witness and par- 
ticipant (Manovich 2001: 207). In this way, with the real-time image generated in the 
time of the viewer, the computer screen becomes a space of performativity. That is, 
rather than emanations of the past, photography has become a performative calcula- 


tion, where temporality is not so much historical as recursive. 


Commercial Conditions: From Film Sales to Algorithms 


Finally, I want to turn briefly to a less commonly discussed aspect of digital pho- 
tography: its underlying economics. Here it is worth beginning with the story of 
Kodak’s failure to capitalise on its innovations in the field of digital photography. 
Already the stuff of legend and myth, the main points in the narrative are clear. 
Kodak’s research laboratories developed the world’s first digital camera in 1975, ina 
team headed by Steve Sassoon. Its development, however, was stunted—confined to 
government clients and limited professional partnerships in the 1980s—because the 
reusable camera storage medium posed a fatal threat to film sales. The threat was 
real, given that film was ‘Kodak’s cash cow’, with an 80% profit margin (DeMoulin 
2009). Raymond DeMoulin, a retired Vice President of the Eastman Kodak Company, 
describes how the company kept to being a ‘film company’ rather than expanding 
into an ‘imaging company’. Thus at the 1982 Photokina show in Cologne, Kodak dem- 
onstrated a display of digitized images, to show its seriousness in digital imaging. 
But it also announced a new film with so-called ‘T’ grain, promising finer grain and 


faster speeds, explicitly ‘to delay the impact of digital’ (DeMoulin 2009: 3). Former 
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Kodak Vice President Don Strickland claims he left the company in 1993 after he 
failed to get backing from within the company to release a digital camera. He told 
BBC News in 2012, ‘We developed the world’s first consumer digital camera and 
Kodak could have launched it in 1992. ... [but] We could not get approval to launch or 
sell it because of fear of the cannibalisation of film’ (BBC 2012). Antonio Perez, the 
most recent CEO of Kodak, has said that the company was still in denial as late as 
2003 (DeMoulin 2009: 14). He recalls that when he started at Kodak, a banner in his 
office read ‘expand the benefits of film’, which he promptly took down, before clos- 
ing eleven of fourteen factories dedicated to film in the next few years. But it was 
too late. Kodak’s stock price declined from US$94 in 1997 to US$16 by 2008 to as 
low as 36 cents before the company filed for bankruptcy in 2012. 

The question is: what has replaced film as the cash cow of consumer photogra- 
phy? Certainly ink and paper are profitable products. But Kodak’s revenues, insofar 
as they exist, are now derived almost exclusively from the patents they own in digital 
research (which continue to feature in almost every camera used today). Meanwhile, 
for the consumer, taking photographs is an apparently cost-free activity, since even 
the purchase of dedicated cameras has become optional now that lenses are embed- 
ded in mobile phones. The fact that the most popular photo app, Instagram, is free 
to download gives us a clue as to what is happening. Photographs have themselves 
become a kind of currency, in that the circulation and exchange of photographic 
images now generates surplus value. On sites like Facebook—who paid US$1 bil- 
lion for Instagram in 2012—the circulation of photographs creates value for share- 
holders through the attraction of users. Of course, algorithms are at the heart of 
this new expropriation of symbolic communicability. The right algorithms are valu- 
able commodities (think of Google’s search engine ranking system). We saw above 
how Google+ are attempting to attract users away from Facebook, by dramatically 
investing in their photosharing software. Flickr’s attempt to patent the algorithm 
for their ‘interestingness’ ranking is instructive, since the ranking of a photograph 
on Flickr is highly dependent on user-supplied metadata such as tagging. Thus, what 
we are currently witnessing is the normalisation of forms of image organisation via 
a combination of human tagging and non-human calculation that are then returned 
back to us in the race for eyeballs. 

As we have seen, it is now abundantly clear that algorithms pertain to the cir- 
cumstances affecting how photography is made, circulated and experienced. This 
is hardly surprising, since algorithms are now integrated into our everyday lives 
more broadly at a variety of other levels, from predictive policing to financial trad- 


ing (Steiner 2012). Algorithms are not in themselves a problem, but they raise new 
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questions about the way decisions are made, issues such as privacy and commercial 
data-mining and the more general commodification of communication itself. This 
latter point is very clearly pertinent to photography, which has always been subject 
to technological developments. Photography’s traditional single-authored logic has 
been supplemented by new conditions that make it more like a social currency, more 
ubiquitous than ever, but most valuable in its aggregations. Given all of this, it has 
recently been argued we need new forms of metaphotography, involving ‘people who 
can figure out effective and timely ways to curate the enormous numbers of images 
online from all sources—amateur and professional alike’, and which ‘contextualizes, 
authenticates, and makes sense of’ the online archive (Ritchin 2013). While this may 
indeed be desirable, such curatorial activities are bound to require computers for 
their success. Indeed, as Boris Groys has argued, curating and digital images are 
bound together: 


Digitalization, that is, the writing of the image, helps the image become 
reproducible, to circulate freely, to distribute itself ... But at the same 
time the digitalized image becomes even more infected with non-iden- 
tity—with the necessity of presenting the image as dissimilar to itself, 
which means that supplementary curing of the image—its curating— 
becomes unavoidable. (2008: 87) 


Algorithms, we might say, are conditioning photography into both particular states 
of existence and desired states for use, and they operate at the speed of light, much 


faster than humans can edit, make sense of, or ‘cure’ them. 
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CHAPTER 8 


Simulated Translucency 


Cathryn Vasseleu 


We perceive translucency as the intermingling of matter and light within media. All 
materials except metals are translucent in varying degrees. Their capacity to dif- 
fuse light inside them has many magical and supernatural associations. Translucent 
substances have an inner glow that can imply interiority, or even the evanescent 
presence of an embodied soul. The ability to emulate these subtleties is a goal of 
photorealistic digital image synthesis. To date the best results have been achieved 
using an appearance modelling approach. Appearance models are algorithms for 
rendering the appearance of all kinds of materials, including those with varying 
degrees of translucency, such as ocean water, tree leaves, milk, marble, paper and 
skin. The aim is to produce a computer model that approximates the appearance of 
an object made of a translucent substance so closely that the photorealistic render- 
ing elicits more or less the same visual response as to a photograph of the physical 
object. What matters in appearance modelling is that the computer model’s formula- 
tion process is not detectable at a sensory level. How then might we begin to aes- 
thetically acknowledge or engage critically and creatively with its synthetic inner 
glow? As will be elaborated, the difference that cannot be seen lies within the sensi- 
bility of simulated translucency.' 

Models for depicting photo-realistic translucency are just one of many light 


rendering techniques, including shadow casting, ray tracing and radiosity. These 
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programs do not simulate the way light is recorded in photosensitive media. They 
simulate the way light rays interact with shapes, material surfaces and with each 
other. Light rendering programs raise questions about how their process can be 
read in the context of multiple codes of realism (pictorial, three-dimensional, pho- 
tographic and cinematic, video, etc.). However, the aspect of photorealistic image 
synthesis that most relates to the sensibility of simulated translucency is that in 
the digitally rendered image the signs of realism are not written in light. Light is 
written in code. 

Appearance models render the appearance of different materials naturalistically, 
based on codified empirical indices of their optical characteristics. Whether simula- 
tions appear realistic, or life-like, because they capture and reconstruct the essen- 
tial physical arrangement of the empirically measurable world, or whether percep- 
tual processes and/or discursive practices determine our experience of simulated 
photorealism, are well-worn topics of debate among scientists, artists and media the- 
orists. In order to consider how we get a model of life-like translucency (both tech- 
nically and as a perceived effect), it is necessary to take a more specific approach. 
This first involves distinguishing translucency from transparency. Transparency 
was pursued as a guiding characteristic and perceived effect of Modernist aesthet- 
ics. Within Modernism and subsequent related movements, translucency has been 
understood in relation to transparency, defined as either an optical or a perceptual 
quality. With alternative ways of understanding the unique consistency of translu- 
cent media in hand we can, in turn, elaborate on the perceived translucency of a 


computer model. 


Differences Between Translucency and Transparency 


A transparent substance is one that is capable of transmitting light as though there 
is no intervening matter. Transparency, understood as a sensibility of both eye and 
mind, is the condition of optical surety and abstract clarity, secured by light passing 
freely though the concrete. As such transparency has aided idealist philosophy in its 
articulation of metaphysical vision, the spiritual soul and metaphoric illumination 
(through a light that illuminates matter but is refractory to incorporation). 

In modern thought, matter’s capacity to transmit light in this way became a key 
factor in the explication of aesthetic concepts in painting, photography, film, sculp- 
ture and architecture. For example the ‘see-through’ effect took a luminous turn 
in art criticism that championed transparency in construction and critical reflec- 


tion. Understood as the experiencing of things ‘being what they are’ (Sontag 1966: 
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13-14), transparency was a liberating, luminous clarity. It applied to minimalist art 
that is open to, and addressed directly to, the transformative exercise of our senses, 
rather than thick with hidden crafting and meaningful content. Transparency was 
also an emblem of the social, corporate and political ideals of structural and opera- 
tional transparency, or openness. I will offer only the briefest sketch of transparency 
here, setting aside discussion of the ‘myth’ of transparency that haunted Modernity 
(revealed in all its opacity by Breton, Benjamin and Duchamp).? 

Architectural transparency was, in its material sense, a guiding construction 
principle of Modernism. At its simplest, a fascination with the space-defining prop- 
erties of transparency was translated into arrangements of glass and other materi- 
als that transmit light in such a way as to be able to see through them, or designs 
that involved unobstructed flows between interior and exterior spaces. The painter 
Robert Slutzky and architect Colin Rowe famously elucidated a distinction between 
the architectural interpretation of transparency in this literal sense, and phenom- 
enal transparency, where transparency is elaborated corporeally and conceptually 
rather than optically (Rowe [1955-56] 1976: 160-62).? As well as being an inherent 
quality of a substance, it could also be a quality of organization. Material explo- 
rations of phenomenal transparency extended the optical characteristic to space, 
or the simultaneous perception of different spatial points within a dynamic field of 
view. The spatio-temporal complexities of phenomenal transparency were (and still 
are) interrogated in the exploration of fluctuating perceptual phenomena such as 
emerging and receding figures and ground, reflections and shadows, and implied 
or revealed surfaces, shallows and depths. Here transparency can apply to opaque 
objects, and to the semi-opacity of translucency. Both are implicated in phenomenal 
interpretations, as levels of transparency. 

With eyes still trained on Modernism and its signature aesthetics of transpar- 
ency, postmodern aestheticians initially focused on the superficiality of simulacra in 
their characterizations of the digital medium. Examples include Baudrillard’s articu- 
lation of the play of appearances of the ‘superficial abyss’ and Brummett’s interroga- 
tion of the ‘prophylactic whiteness’ or occluded interior of electronic images, devices 
interfaces and systems (Darley 2000: 58-77). Transparency and all it signified was 
obliterated by the computer interface according to such analyses. However the qual- 
ity continued to figure as a key feature of ‘Supermodernism’, which conceives of 
architectural space as an ‘empty medium’, purposefully striving for transparency 
(Hans Ibelings 1998). Rather than communicating a message, supermodernist archi- 
tecture works like an invisible, enveloping medium that creates a unique sensory 


experience. 
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Lev Manovich (2007) extends this aesthetics to the dissolution of personal tech- 
nological devices into ambient objects in the era of ubiquitous computing. Machines 
disappear as they become implanted inside other objects, materials and surfaces. In 
Manovich’s analysis transparency and translucency are regarded synonymously as 
the aesthetic signature of the boundless flexibility of the computer as an all-in-one 
meta-machine. On closer examination, his analysis relies on a difference between 
these two qualities. The transparency of the enveloping medium requires qualifica- 
tion of some kind, insofar as its aesthetic signature is not abysmal emptiness, but a 
comforting, sensual, enveloping shell. It is translucency, understood as a medium’s 
capacity for incorporation and diffusion of other media, which metaphorically aids 
product designers’ articulation of the computer as an embracing, universal simula- 
tion machine. 

This brings us to an aspect of phenomenal transparency that is taken for granted. 
Its transparency cannot be separated from the order of matter, or a translucent 
materiality. The tendency of light to thicken and matter to dematerialize in translu- 
cency is associated with the appearance of a dynamic material universe. This asso- 
ciation is made and developed in philosophical accounts of perception by Maurice 
Merleau-Ponty and Henri Bergson. 

Merleau-Ponty gives a unique inflection to the partial opacity of translucency. 
He regards matter as the invisible condition of seeing rather than an obstacle to 
perception. For example, looking into an ornamental garden pool and observing the 
tiling clearly visible below, Merleau-Ponty reflects: ‘I do not see it despite the water 
and the reflection there; I see it through them and because of them’ (Merleau-Ponty 
1962: 182). Here, the tiling materializes for the viewer in the intervening thick- 
ness of water: ‘If there were no distortions, no ripples of sunlight, if it were with- 
out this flesh that I saw the geometry of the tiles, then I would cease to see it as it 
is and where it is—which is to say, beyond any identical, specific place’ (Merleau- 
Ponty 1962: 182). Merleau-Ponty’s account of visual perception is anti-optical. It runs 
counter to a representational view of the empirical world, or a merely physical-opti- 
cal relation in which vision is based on optical surety. For Merleau-Ponty the ‘see 
through’ quality of transparency is related to a translucent materiality, not immate- 
riality. What transparency means for Merleau-Ponty is the ability to see through the 
intervening thickness of media. 

While Merleau-Ponty equates transparency with a voluminous translucence, he 


understands the dynamics of translucency in terms of mirroring: 
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every object is the mirror of all the others ... 1 can ... see an object in so 
far as objects form a system or a world, and in so far as each one treats 
the others round it as spectators of its hidden aspects and as guaran- 
tee of the permanence of those aspects. (1962: 68-69) 


Thus reflected, translucency is the condition of the apprehension of the visible as a 
system or universe of beings that disclose themselves in and to each other as a whole 
(a world of implicated figures and grounds explored in phenomenal elaborations of 
architectural transparency). The visible is not a formless multiplicity of perceived 
objects-in-general that are foremost given over to abstraction and clarification. Any 
seeing of an object is a participation in the translucence, that is, the whole-in-one 
aspect of things: ‘the completed object is translucent, being shot through from all 
side by an infinite number of present scrutinies which intersect in its depths leav- 
ing nothing hidden’ (Merleau-Ponty 1962: 68-69). The ‘thickness’ of translucence 
is not a geometrical dimension, as refined in the perspective painting techniques 
of the Renaissance to facilitate a more exact, artificial construction of the world. 
What is gathered in the intersecting voluminous spacing of translucency is the 
dynamic unfolding of the fabric of the world. This is conveyed in the uncontained, 
infinitely dispersed radiation of the visible, whose freedom in self-arrangement can 
be described as ‘internal animation’. 

Merleau-Ponty describes translucence as the shimmering clarity of a phenom- 
enological vision anchored in the material world, but a simulation of translucent 
appearance is an image calculated by a disembodied model. Here translucency has 
a separate, non-phenomenological existence in a virtual form that is more readily 
aligned conceptually with Bergson’s understanding of the dynamics of perception. 
Like Merleau-Ponty, Bergson does not regard perceptual experience as a function 
of a physical-optical relation between a subject and a transcendentally illuminated 
object-world. Instead, the appearance of the material universe is derived through 
our perception. In Bergson’s account, material objects have independent functions 
and real actions, but a living body participates in creating perceptions through the 
momentary delimitation and isolation of a light that would otherwise pass by and 
remain unrevealed. The world of perception manifests its spontaneity insofar as: 
‘Images detach from themselves that which we have arrested on its way, that which 
we are capable of influencing’ (Bergson 1988: 37). In this schema, a perception is 
a phenomenon of the same order as a virtual image. It is an effect of light that has 
been impeded and arrested momentarily, reflected without otherwise affecting the 


totality and freedom of action of matter. Reflections can appear as isolated moments 
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of intentionality (contracted by and limited to our interest), as when images ‘appear 
to turn towards our body the side, emphasized by the light upon it, which interests 
our body’ (Bergson 1988: 36). In this way images can be perceived by us as living 
matters, or exist and act independently without being related to an individual, illu- 
minating consciousness. 

Bergson sets out in his vitalist philosophy to dismantle the notion of an immuta- 
ble universe, which he associates with an intellectual viewpoint that renders matter 
inert by defining it in terms of properties that accentuate its materiality (Bergson 
1911: 202). By way of contrast, he describes consciousness as a translucence in 
which the eye is elementally in things, universalizing a pure, non-human vision in an 
a-centred universe of images. Bergson’s mobile, iridescent consciousness is a trans- 
lucence that dematerializes solidified matter and animates the universe of images 
and objects. The chaotic interaction of virtual influences is the source of movement 
in both these things. 

Both Merleau-Ponty and Bergson turn to the properties of translucent media to 
think their philosophies of perception through. Both also invoke the dynamics of 
multiple mirroring when elaborating on a global translucence. On closer consider- 
ation that association needs further elaboration. Mirrors are opaque structures that 
reflect light without altering its clarity. Reflection alone does not account for the 
qualified clarity of translucency. This raises another issue: the unique characteris- 
tics of translucency that distinguish it from transparency. 

Translucency is an incomplete transparency, or partial opacity. It is a liminal 
quality, existing on the threshold between clarity and obscurity. The ‘inner light’ 
characteristic of translucency has a thickness that is related to material consis- 
tency, not spatial depth. Optically, a translucent substance is one that is capable 
of transmitting light, but also causes sufficient diffusion of that light to prevent the 
perception of clear and distinct images through it. Instead of being entirely perme- 
able to light, as one sees things through a clear glass window, a translucent sub- 
stance is suffused throughout with a shadowless, diaphanous lucidity. The passage 
and containment of light creates, simultaneously, opacities in our field of vision.* We 
can see only partly, not fully, through translucent matter. Light is reflected within 
the material before being either absorbed or transmitted. The internal scattering of 
light gives rise to a soft appearance, rather than a luminous opening. 

In the stained glass windows of Gothic cathedrals, translucency was used to cre- 
ate the impression of the unveiling of veiled truths. The iridescent beauty of par- 
tially opaque panels at once shrouded and revealed an ineffable, divine light (Von 
Simson 1962: 120-22). Elsewhere the ambivalence between opaque surface and 
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transparent opening has been employed to create a concept of a multi-natured fluid 
space. Translucent paper screens, a characteristic feature of Japanese architecture, 
were used in this way long before the concept was adopted by Western architec- 
ture in the early twentieth century. Presenting a glowing surface when closed, the 
screens allow direct access to nature when open (Kaltenbach 2004: 7). 

Artists and architects manipulate the mutable characteristics of translucent 
materials to create a ‘tactile’ sensibility, or a sensual play between surfaces that 
admit light and the smooth, soft appearance of inner substance. Involving a sense of 
matter as much as light, translucency gives luminous contours to the material pas- 
sages of inner-space. These passages are not rendered invisible or obliterated, as 
they are in optical transparency. Light blurs as it is incorporated and moved about 
within the blurring substance. Illumination takes on a sensuous character, not the 
clarity seen in the passage of a metaphysical light through material entities. If light 
figures as a connection between spirit and matter in virtually every culture, in trans- 
lucency, it remains ambiguous whether light passes through or is moved by matter. 
While unobstructed openness is a defining characteristic of transparent relations, 
there are interruptions of clarity in the opaque/transparent glow of translucency, 


where matter has the power to admit, contain and transmit light. 


A Computer Model of Translucent Media 


In 2001 Henrik Wann Jensen and his colleagues at the University of California, San 
Diego, published a groundbreaking technique which made it possible to emulate the 
subtle characteristic traits of perceived translucency, such as colour bleeding within 
materials and diffusion of light across shadow boundaries (Jensen et al. 2001). A 
model that works for shiny, reflective materials such as metals assumes that light 
makes contact with and leaves the surface at the same position. In order to work for 
translucent materials the model must assume that light rays alter their position in 
their transmission through intervening materials, leaving the surface at a different 
point. These include liquid, solid and gaseous substances whose internal structure 
and composition scatter light, thus contributing to their visual complexity. 

Optically, a significant amount of light is transported below the surface in trans- 
lucent substances. Computer-models of translucent materials look completely wrong 
without the inclusion of this characteristic, known as subsurface scattering. The 
technique Jensen’s team proposed made the inclusion of this characteristic practi- 
cal. It worked by approximating the subsurface scattering of light diffusing through 


translucent materials, without having to laboriously trace the individual photons.°® 
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Jensen describes his photon mapping 
technique as the recovery of the optical 
properties of matter. From this descrip- 
tion, are we meant to understand that 
the likeness distilled in Jensen’s model 
refers to a refractory substance known 
only by its empirically defined con- 
tours? Surely we grasp the appearance 
of an implied thickness and softness in 
a more substantive way—by responding 
to the image in physical terms, under the 
direction of the model’s revision of that 


arrangement? Rather than posing an 


Figure 1. Three glasses of milk. From left to right: skim milk, whole 
milk, and diffuse milk. The skim and whole milk have been rendered 
using a model that takes subsurface light transport into account. of a substantive form of translucency 
The diffuse milk has been rendered using a model that defines 
how light is reflected at an opaque surface, which results in a hard 


appearance, making the milk look more like white paint rather than in such a way that viewers are corpore- 
like milk (Jensen 2006). Courtesy of Henrik Wann Jensen. 


abstract identity, the life-like simulation 


hinges on the elaboration of an image 


ally attuned to its material character. 

Jensen’s team knows from experience 
that their task involves using a digital medium to render a materially determined 
‘inner light’ in a visually interpretable way. 

They know for example, that it is not solely the proportion of light and dark in 
an image, but their spatial relationships that make an object look translucent. They 
know that translucency affects the spatial structure of images; blurring detail, etc. 
Most importantly, they know that spatial organization alone does not tell us whether 
an object is translucent or not. Our vision is attuned to other factors unrelated to 
spatial layout, which influence perceived translucency (Fleming et al. 2004). 

Jensen’s team tests their algorithm’s approximation of translucency by compar- 
ing the model’s appearance with a physical model-box containing equivalent vol- 
umes and surfaces in a substantial form.® The visual similarity is taken as proof of 
the algorithm’s ability to synthesize an image of an object whose translucent appear- 
ance is indistinguishable to sight from a physical translucent object. We read the 
object’s material character from how translucent it appears in the context of its sur- 
roundings. Milk, for example, is a translucent substance that Jensen’s model can 
replicate. Synthesized in such a way that we can identify the familiar visual surface- 
texture of physical translucency, a substance looks milk-like rather that looking like 
white paint (Jensen 2006). 
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Semioticians ascribed an indexical role to photography, understood as the rep- 
resentation of idealized physical entities. Here indexicality refers to a physically 
enacted connection between an object and its traces in a photographed image. 
Computer modelling reinvents the age-old trick of seeing by mimetic illusion. The 
type of index that enters into the computation of translucent appearance is the 
‘index of refraction’, along with other parameters necessary to derive a computer 
model for a particular optical property. It is possible to devise models for handling 
all kinds of translucent substances, including the model already referred to, that 
can compute the parameters of milk-like translucence. Having the capacity to be fed 
descriptions of the fat and protein content of various milk products, it can handle the 
translucence of skim, regular, full fat, etc. 

The experience we have of this model is regulated by an already worked-out 
make-up that is judged by the extent to which it achieves the intended effect; that is, 
a seamless visual approximation of familiar objects. Appearance modelling is based 
on the twin premise that images appear realistic when the model can capture, incor- 
porate and reconstruct the essential optical-physical arrangement of the empirical 
world, modelled in accordance with psychophysical measurements of human powers 
of vision. In actuality, the model is an optical description that is designed for viewers 
to ‘fill in’ (guided by a prior carnal knowledge of milk’s translucency) rather than a 
purely optical description. 

As a circumscribed visual rendering, the image is only a partial representation. 
An algorithm for simulating milk-like translucency is not indicative of the mastery 
of phenomenal complexity. It is indicative of the mastery of the model. A model cre- 
ates a unique perspective that tantalises and seduces. For example, we might expe- 
rience milk in the Coca-Cola model way, as a product positioned in the company’s 
global marketing strategy. In 2001 Coca Cola proposed a new line of dairy drinks 
for children under the working title ‘Project Mother’ (Brown 2001; Stevenson 2002). 
‘Milk’ was not conceived of primarily as a drink made of milk. It was a beverage 
that was designed to capture the breakfast market because its ingredients would 
be judged sufficiently nutritious to win maternal approval. In other words, Project 
Mother wagered that appearing to be nutritious would be more seductive than being 
the real thing. 

A computer model of a light-scattering substance is a psychophysical approxima- 
tion that obviously doesn’t affect our senses in the way a glass of milk does. We do 
not encounter the semblance as we would an ‘authentic’ glass of milk, whose mate- 
rial character is expressed in its countless corporeal manifestations; its coldness if 


it has been poured from a container in the fridge, forming a skin if it is hot, leaving 
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a white moustache on our upper lip, tasting sour because it has gone off, giving us 
succour as did our mother, making us sick if we are lactose-intolerant. 

Just as Alfred Hitchcock is able to render milk’s characteristic translucency ina 
suspect way in the film Suspicion (1941), turning its glowing interior into something 
equally viewable as a poisoned brew,’ the computer model formulates its own unseen 
parameters of sensibility. It allows us to see exactly (enough to distinguish between 
skim and whole milk) in adherence with its abstracted optical-physical reality. This 
is the ‘inherently false’ reality of a formal arrangement that cannot be realized phys- 
ically. We cannot experience the material nature of milk based on how exactly the 
model enables us to understand its milky look.’ Jensen’s model is a distillation of 
translucent appearance, generated by the actions of an automated, virtual light. The 
paths of virtual photons are calculated to lose and change position between enter- 
ing and leaving the simulated material, thereby reflecting its internal structure and 
material makeup. In the interstices, light ceases to be a radiant beam of energy and 
becomes something more akin to an animated, ambient light, indistinguishable from 
and informative of its digital-milk environment. The characteristic optical behaviour 
occurs spontaneously in photon mapping but its nature is pre-scripted. The degree 
of luminous ‘density’ and ‘bleeding’ is calculated in advance by an invisible structur- 
ing medium that determines the image’s translucent appearance. In the modelling 
process, the invisible ordering of a materially given, luminous sensuality is recast 
as a sensuality of automated light flows. Jensen activates the flow to approximate 
a particular visual effect. His technique of image synthesis contains the chaos of a 
physical universe within its complex mapping of light’s movement by matter. 

Despite the model’s artificially imposed order it is not disorienting to a viewer. 
Translucent appearance is instantaneously perceived, albeit in terms of new struc- 
ture. The simulation amounts to a formal re-arrangement of the visible based on 
the indices of ‘subsurface scattering’; that is, a structure determined by codified 
empirical indices of optical characteristics. Thus the model creates its own invisible 
parameters of complexity (complex variations in the translucent appearance of dif- 
ferent materials). An ordered appearance of soft, smooth diaphanousness is sponta- 
neously computer-generated, but is based on a quantifiable light-scattering function 
that must be calculated in advance. Altering the variables can produce unexpected 
effects, but once scripted the actual behaviours of virtual photons interacting with 
virtual surfaces is fixed. As well as being motivated to act in a formally specified 


way, the random scattering of light is a calculated variable. 
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. Figure 2. Different skin types simulated with model (top) compared 
render the appearance of human skin, to actual photographs of real skin samples (bottom) (Donner and 


which is a multilayered translucent sub- Jensen 2006). Courtesy of Henrik Wann Jensen. 


Simulated Human Skin and Its 
Perceived Translucency 


Appearance models describe the proce- 
dures for animating the variables that 
apply to the characteristic appearance 
of materials and media. Jensen’s model 


for calculating subsurface scattering 
was quickly adopted by the 3D computer 
animation industry to photo-realistically 


stance. These multiple layers scatter 

light differently according to their different composition, resulting in a reddish cast 
to which human vision is finely attuned. More than with other materials, the slight- 
est simulation errors are noticeable in human skin (Jimenez et al. 2010: 32). Just 
as the milk model can render the appearance of different types of milk (whole and 
skim etc.), the variables for modelling human skin can be altered to render shades 
of difference in its translucent appearance. The prototype for skin (skin1) was mod- 
elled, not surprisingly, as a facial close-up of a pale pink translucent cheek and 
fleshy, lipstick-reddened lips.? The modelling of various other skin-shades followed: 
Caucasian, Asian and African (Donner and Jensen 2006). 

While the realistic appearance of simulated milk does not strike an uncanny 
note, the photorealism of a computer-generated actor can. The simulation of the 
‘inner light’ occluded in human facial-skin initially exposed something of the pre-for- 
mulated ‘soul’ occluded in the polygonal faces of digital humans. These characters 
stumbled into what Masahiro Mori (1982) calls the ‘uncanny valley’. Here, instead of 
completely identifying with a figure that appears human, audiences are repelled by 
traces of its robotic nature. Rather than knowing what they are looking at, viewers 
are disturbed by a presence they are not quite able to identify. When synthespians 
first appeared in 3D animations, audiences scoured their faces for traces of hid- 
den engineering.'° Viewers were forced to adopt an interrogative mode of perceiving 
while watching the moving images. Their vision was moved by practical concerns, 
born of an uncertainty about how they should view the photorealistic approxima- 
tions of cinema screen actors they were seeing. 

The simulation of human facial-skin announced a particular way of seeing the 
world, and with it a novel way of being (in both an existential and social sense). It 
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announced a new form of animated presence; simulated matter with its own built- 
in, manipulatable, photorealistic character. It also announced new cultural image- 
forms—simulated photorealistic Caucasian, Asian, African characters that posed 
automated racial identities with their own optically defined order. 

Simulation models do not replicate the natural world. They demolish a natural- 
ized metaphysical viewpoint and replace it with systems that generate a world-order 
composed of quantifiable, manipulatable results. The model for subsurface scatter- 
ing results in a controlled appearance of inner light that is indicative of different 
types of material character. An ambiguous quality is digitally recast as a precisely 
differentiated spectrum of ‘signature’ translucencies. Disclosed according to this 
methodical replacement of a naturalized metaphysical viewpoint, translucency is 
the automatic outcome of a model with adjustable parameters that renders an empir- 
ical description as believable as the appearance of photographed physical objects. 

Twentieth century avant-garde photography and cinema practices confounded 
indexical readings of their art by privileging material process. Critically-oriented 
digital-based media practices have progressively confounded readings of their art 
in terms of reified concepts of immateriality, abstraction and materiality. We have 
yet to discover procedures that render the fixed behaviour of each shade of material 
character seen in photorealistic translucency in a questioning way. For this we need 
alternative ways of encountering simulated lighting effects, apart from blindly incor- 
porating the perspective of the computer model. It takes more radical approaches to 
the process of approximation to shake our faith in the ‘recovery’ process of appear- 
ance models." 

For the moment experimentation with light rendering programs has been lim- 
ited to extending their applications and aesthetic possibilities. Appearance models 
can offer an alternative perspective to naturalistic depiction. Architectural design is 
increasingly applying digital lighting programs to choreograph spaces that address 
human perception and multi-sensory experience, more in line with the aesthetics 
of Supermodernism. Here experimentation with computer-generated visual effects 
aims to evoke a sensuously rather than optically defined spatiality. 

Within the computer graphics industry, refinements in light rendering tech- 
niques serve an essential role in overcoming undesirable uncanny effects in 3D com- 
puter animation. However, subsurface light transport models are costly in render- 
ing time and are not suited to computer games and many graphics formats. Recent 
experimentation with alternative solutions that can be rendered quickly in games 
environments includes algorithms which translate the scattering effect simulated 


in a 3D model to screen space in a simplified form (Jiminez et al. 2010), and the 
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proposal of even more rapid shading models (Gomez 2011). Unlike photon-mapping 
models of subsurface light transport, shading models provide impressionistic visual 
cues such as light source direction, colour gradients and blurriness, that help make 
an object appear translucent. With experimentation to produce ever faster ‘real-time 
translucency’ comes a proliferating technical typology of synthetic illumination. 
Although superior in terms of speed, the look achievable with real-time techniques 
is still inferior to the results achievable with photon-mapping techniques. Within the 
typology of synthetic illumination the subsurface light transport models discussed 
throughout this essay produce images that are formatted with the attributes of ‘true 
translucency’. 

Here we have not only a computational model but also a model form of trans- 
lucency. Ultimately, the unparalleled authenticity of ‘true translucency’, as recog- 
nized in computer graphics parlance, is a revealing expression that encapsulates 
the unseen substrate of digital image synthesis: an all-in-one translucent controlling 


medium, with an infinite capacity for absorption and diffusion of other media. 


References 


Bergson, Henri. (1908) 1988. Matter and Memory. Translated by Nancy M. Paul and 
W. Scott Palmer. New York: Zone Books. 


———. 1911. Creative Evolution. Translated by Arthur Mitchell. New York: Henry 
Holt and Company. 

Brown, Alecia. 2001. ‘Coca Cola looks to market milk.’ ABC Rural News, 4 January. 
http://www.abc.net.au/rural/news/stories/s230115.htm. 

Darley, Andrew. 2000. Visual Digital Culture: Surface Play and Spectacle in New 
Media Genres. London: Routledge. 

Deleuze, Gilles. 1986. Cinema 1: The Movement-Image. Translated by Hugh 
Tomlinson and Barbara Habberjam. Minneapolis: University of Minnesota Press. 

Donner, Craig, and Henrik. W. Jensen. 2006. ‘A spectral BSSRDF for shading human 
skin.’ Eurographics Symposium on Rendering: Cyprus, June: 409-17. 

Fer, Briony. 2000. ‘Some translucent substance, or the trouble with time.’ In Time 
and the image, edited by Carolyn Bailey Gill, 69-78. Manchester: Manchester 
University Press. 

Fleming, Roland W., Henrik W. Jensen, and Heinrich H. Bulthoff. 2004. ‘Perceiving 
translucent materials.’ In Proceedings of the 1st Symposium on Applied 


Perception in Graphics and Visualization, APGV 2004, 127-34. Los Angeles, 
California. 


175 


176 Cathryn Vasseleu 


Frisvad, J. R., N. J. Christensen, and H. W. Jensen. 2007. ‘Computing the scattering 
properties of participating media using Lorenz-Mie theory.’ In ACM SIGGRAPH 
2007, Article 60, 1-10. New York: Association for Computing Machinery. 


Gomez, Christian Gascons. 2011. ‘Fast Simulation of Translucent Objects.’ Master in 
Computing diss., Polytechnic University of Catalonia. 


Harbison, Robert. 1997. ‘Models.’ In Thirteen Ways: Theoretical Investigations in 
Architecture, 84-98. Cambridge, Mass.: MIT Press. 


Ibelings, Hans. 1998. Supermodernism: Architecture in the Age of Globalization. 
Rotterdam: NAi Publishers. 


Jensen, Henrik Wann. 2001. Realistic Image Synthesis Using Photon Mapping. 
Wellesley, Mass.: AK Peters. 


———. 2005. ‘Cornell box images.’ Last modified 30 November, 2005. http:// 
graphics.ucsd.edu/~henrik/images/cbox.html. 


——-—. 2006. ‘Subsurface scattering.’ Last modified 16 April, 2005. http://graphics. 
ucsd.edu/~henrik/images/subsurf.html. 


Jensen, Henrik Wann, and Juan Buhler. 2002. ‘A rapid hierarchical rendering 
technique for translucent materials.’ ACM Transactions on Graphics, SIGGRAPH 
2002, San Antonio, 21 July (3): 576-81. 


Jensen, Henrik Wann, S. R. Marschner, M. Levoy, and P. Hanrahan. 2001. ‘A 
practical model for subsurface light transport.’ Computer Graphics: Proceedings 
of SIGGRAPH 2001, 511-18. New York: Association for Computing Machinery. 


Jiminez, Jorge, David Whelan, Veronica Sundstedt, and Diego Gutierrez. 2010. ‘Real- 
Time Realistic Skin Translucency.’ IEEE Computer Graphics and Applications, 
July/August: 32-41. 

Kaltenbach, Frank, ed. 2004. Translucent Materials: Glass, Plastic, Metals. Basel: 
Birkhauser Publishers for Architecture. 


Manovich, Lev. 2007. ‘Information as an aesthetic event.’ Lecture given at the 
Tate Gallery, London, 8 September. http://manovich.net/index.php/projects/ 
information-as-an-aesthetic-event 


Merleau-Ponty, Maurice. 1962. The Phenomenology of Perception. Translated by 
Colin Smith. London: Routledge; Kegan Paul. 


Mori, Masahiro. 1982. The Buddha in the Robot. Tokyo: Charles E. Tuttle Co. 


Rowe, Colin. (1955-56) 1976. ‘Transparency: literal and phenomenal (with Robert 
Slutzky).’ In The Mathematics of the Ideal Villa and Other Essays, 160-83. 
Cambridge, Mass.: The MIT Press. 


Sontag, Susan. 1966. ‘Against interpretation.’ In Against Interpretation and Other 
Essays, 4-14. New York: Farrar, Strauss & Giroux. 


Simulated Translucency 177 


Stevenson, Seth. 2002. ‘Things go better with ... a shelf-stable kids’ lactic drink.’ 
Sydney Morning Herald, 18 May, 43-47. 


Vasseleu, Cathryn. 2003. ‘What is virtual light?’ Culture Machine 5. http://www. 
culturemachine.net/index.php/cm/article/view/254/237. 


———. 2012. ‘Material-Character Animation: Experiments in Life-like 
Translucency.’ In Carnal Knowledge: Towards a New Materialism Through the 
Arts, edited by Estelle Barrett and Barbara Bolt, 153-69. London: I.B. Tauris. 


Vidler, Anthony. 1992. ‘Transparency.’ In The Architectual Uncanny: Essays in the 
Modern Unhomely, 217-25. Cambridge, Mass.: The MIT Press. 


———. 2003. ‘Transparency: literal and phenomenal.’ Journal of Architectural 
Education 56 (4): 6-7. 


Von Simson, Otto G. 1962. The Gothic Cathedral: Origins of Gothic Architecture and 
the Mediaeval Concept of Order. 2nd ed. New Haven: Princeton University Press. 


Weschler, Lawrence. 2002. ‘The race to build a perfect face.’ Wired, 6: 120-27. 


Notes 


1. A portion of this essay has appeared in ‘Material-Character Animation: Experiments 
in Life-like Translucency,’ published in Carnal Knowledge: Towards a New Materialism 
Through the Arts, edited by Estelle Barrett and Barbara Bolt (Vasseleu 2012). I 
originally wrote the extract material as a sequel to an earlier essay, ‘What is Virtual 
Light?’, which considered the development of the computer as a light-source (and 
light rendering programs specifically) in terms of a broader, cultural refiguration 
of light’s ontological status (Vasseleu 2003). The sequel essay went unpublished 
until an opportunity arose to expand on the material process of computer-rendered 
translucency in the longer essay published in Carnal Knowledge. I thank Barbara Bolt 
and Estelle Barrett for agreeing to the republication of the extract in this volume 
dedicated to critical accounts of contemporary digital light-based technologies. 


2. The ‘myth’ of transparency and its revival as a new principle of ‘good modernism’, 
exemplified by making a monument disappear in relation to its context (as in the 
Louvre Pyramid), as well as and other aspects of architectural transparency are 
discussed by at length by Anthony Vidler (Vidler 1992: 217-25). 


3. Robert Slutzky’s influence as a painter on the idea of phenomenal transparency 
in architecture is also discussed by Vidler (Vidler 2003: 6-7). Rowe and Slutzky 
acknowledge Gyorgy Kepes’ initial description of phenomenal transparency, which 
they elaborated in their essay. 


4. A shutter closing, functions as a thickening or blind spot that can be seen though. The 
closed off inner core of translucent substances functions likewise. Realised in both 
form and substance, the eclipse is a temporary death or moment of detachment of the 
subject, partially suspended from the field of vision (Fer 2000: 77-78). 
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5; 


10. 


11. 


For a full account of the process of photon mapping see Jensen’s book on the subject 
(Jensen 2001). 


The Department of Computer Graphics at Cornell University is home of the Cornell 
Box, which is a physical model housed at Cornell University, and also the ubiquitous 
test scene for global illumination. Since 1984 it has been used in the development 
and testing of algorithms for rendering light naturalistically, through physically 
based lighting models and perceptually based rendering procedures. The aim of 
such rendering methods is to produce images that are visually and measurably 
indistinguishable from real world images. 


Deleuze singles out this famous scene in Alfred Hitchock’s Suspicion (1941) as an 
instance in which cinema creates a ‘mental image’ by making relation itself the object 
of an image. The appearance of a thing may relate to the thing itself in a familiar, 
natural way, or the same appearance might leap out and strike a contradictory note 
(Deleuze 1986: 203). A computer model creates an appearance that may relate to the 
thing itself in a familiar, natural way, or not, when the rendered image calls the thing 
that it relates to into question because, for instance, the appearance of a glass of milk 
accidentally looks like a glass of white paint. 


I am referring here to the beguiling perspective the model conjures, as discussed 
for example by Robert Harbison (1997: 84-98). There are other grounds for 
incommensurability, such as that between the model’s abstracted lighting and a 
situated, phenomenal ‘lighting’ which invisibly constitutes the visible (Merleau-Ponty 
(1962: 311). The translucency of digital milk presents to us as a technically accessible, 
machine rendered likeness, which is produced by data that is unintelligible except in 
relation to human-machine systems. 


The recovery of the optical properties of ‘skin1’ is indicative of one of the fantasy 
pursuits that appearance modelling can serve; in this case a familiar formal aesthetic 
endeavour involving a desired love object (that doesn’t resemble Wall-E’s gleaming 
white girlfriend). 


At least one reviewer was unconvinced that Jensen’s formulae could ever master the 
subtle intricacy with which light radiates from inside the face, that is, the so-called 
‘inner light’ of consciousness (Weschler 2002: 120-22). 


I take this issue up in relation to simulated translucency by assessing its life-like 
quality in comparison to Rachel Whiteread’s tactile rendering of translucent cast 
resin sculptures. Whiteread works by abstraction, solidifying unseen spaces to 
produce uncanny perspectives that otherwise elude perception and thought. In the 
case of a simulation, this might include recasting the flow of virtual light to show that 
the material character of the world it models is returned to us strangely altered in its 
formal exactness (Vasseleu 2012). 


CHAPTER 9 


Mediations of Light: Screens as Information 
Surfaces 


Christiane Paul 


This chapter investigates the digital mediations of light enabled and supported by 
screens as information surfaces. As media archeologist Erkki Huhtamo, in particu- 
lar, has pointed out, screens need to be examined not only as designed artefacts, but 
also with regard to their uses, their intermedial relations with other cultural forms, 
and the discourses that have framed them in different times and places. In order to 
investigate these aspects, one needs to first reconsider relationships between the 
display screen and projection, which have become more complex in the age of digital 
media, as well as the encoded nature of the digital image. Digital media also have 
introduced a shift from mostly ocularcentric paradigms—privileging vision as the 
basis for sense making—to a combination of vision and enacted embodiment that 
often employs light and shadow as enabling principles. Using these considerations 
as a starting point, this text explores screens in new media in their multiple roles 
and functionalities—as windows, mirrors, and membranes or as watchful, reactive, 
and transposing interfaces—to determine their role in the mediation of digital light. 
Light is considered in its relation to temporality and movement and as a crucial ele- 


ment in mediating embodied action. 
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Light / Screen / Projection 


Plato’s famous ‘Allegory of the Cave’ describes the existential predicament of a 
group of prisoners trapped in a cave: a gigantic fire at the cave’s entrance sepa- 
rates the prisoners from the outside world while a wall in front of them provides 
the ‘screen’ for the only image of reality they know—the shadow play created by 
the people moving behind them, in front of the fire. Essentially, Plato’s allegory is 
a story of enlightenment, of the ability to grasp the invisible truths underlying the 
apparent surface and reflections of the world surrounding us. At the same time, the 
allegory of the cave captures the essentials of mediation and the construction of 
imagery, the relationship between light and shadow, the screen and projection (in all 
of its meanings). 

If one would update Plato’s allegory for the age of cinema, one might replace the 
cave with a movie theatre, the fire with a projector, and the wall in front of the pris- 
oners with a screen. In the age of digital media and ‘digital light’, the relationships 
between light, shadow, screen, and projection have become more complex: light is 
generated by electronic ‘on and off’ states, shadows can be ‘read’ and recoded to 
‘move’ elements projected on a screen, and the relationship between screen and pro- 
jection has shifted. Commenting on the relationship between screen display vs. pro- 
jection, Steve Dietz poses the question: 


Isnt any similarity—or difference—in the screen, not the projec- 
tion, which is invisible, essentially, when projected from the rear? 
Ultimately, except for the philiacs among us, the difference between 
35mm front projection and TV is not about throw distance and trajec- 


tory. It is about scale and resolution. (2006) 


To address qualities of the digital screen and projection, one needs to consider the 
role that light plays in rendering imagery visible. Whether there are distinctly dif- 
ferent qualities between natural light and the light emitted by digital devices is 
beyond the scope of this text (and is addressed by other chapters in this book). In the 
digital age, what is visible on a screen, be it the computer screen or the wall space 
onto which a digital image is projected, is controlled by modulations of light in a 
spatially mapped array of bits or pixels. As opposed to the opto-mechanical film pro- 
jector, which runs film frames through which light is projected, digital LCD (Liquid 
Crystal Displays) or DLP (Digital Light Processing) projectors create imagery from 
arrays of pixels. In the LCD projection (or screen) each pixel typically consists of a 
layer of molecules aligned between two transparent electrodes, and two polarizing 


filters. In DLP projectors, the image is created by microscopically small mirrors laid 
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out in a matrix on a semiconductor chip, with each mirror representing one or more 
pixels in the projected image. Each mirror can be repositioned rapidly to reflect 
light either through the lens or onto a so-called heat sink or light dump. Mirrors are 
toggled between these two orientations or on and off states. 

Another aspect of the digital image that is, at least indirectly, relevant to discus- 
sions of digital light is its encoded nature, which is commonly juxtaposed against 
the indexical one of photography and film. When Joseph Nicéphore Niépce created 
what is now commonly referred to as the first ‘photograph’ in 1826, a view from his 
workshop window using asphalt on a pewter plate, exposure to light was the core 
element in creating an image with an indexical relationship to its surroundings. One 
might argue that indexicality is not completely abandoned in a digital photograph. 
The fact that the object potentially could be completely ‘simulated’ in the digital 
medium does not turn the digital photograph into pure iconicity. While the digital 
photograph does not rely on light bouncing off an object to ‘imprint’ an image onto 
emulsion, it still encodes the light reflecting off a physical object. In the case of a 
digital image this process of encoding occurs during creation, editing, and distri- 
bution via a digital system (from computer screen to projection). German photog- 
rapher Andreas Miller-Pohle’s project Digital Scores (Digitale Partituren, 1995-98) 
nicely illustrates this encoded nature. In Digital Scores III Muller-Pohle translated 
Nicéphore Niépce’s photograph into digital information. Miuller-Pohle digitized 
the image and then translated the seven million bytes into alphanumerical code, 
which was distributed onto eight square panels. The information transcribed onto 
the panels remains undecipherable yet contains an accurate binary description of 
the original. Digital Scores both points to the fluid transition of information in the 
digital realm and to the different forms of encoding inherent to the digital and pho- 
tographic medium. The encoding process itself, rather than the representational 
qualities of the image, becomes the central point of the work. The possibilities of 
encoding offered by the digital image play an essential role in enabling new configu- 


rations in the exploration of light in its relationship to the (projection) screen. 


Ocularcentrism vs. Body Schema 


Jacques Derrida suggested that the dichotomy of darkness and light is the founding 
metaphor of Western philosophy: ‘the entire history of our philosophy is a photol- 
ogy, the name given to a history of, or treatise on, light’ (quoted in Vasseleu 1997: 
5). One could argue that presence, being, ontology, rational knowledge, enlighten- 


ment, epistemology etc., at least on some level, are all tied to notions of light and 
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vision. As linguists and theorists such as Martin Jay have pointed out, the English 
language semantically connects vision and the eye with the Cartesian cogito and 
rational knowledge in the statement ‘I see’. 

The link between vision and interpretation of the world is central to the concept 
of ocularcentrism, which Martin Jay popularized in his book Downcast Eyes: The 
Denigration of Vision in Twentieth Century (1993). Jay argues that there is no ‘vision’ 
or sight prior to cultural mediation, since everything we see is shaped through our 
cultural and historical context. Nevertheless ocularcentrism—which poses an objec- 
tive world independent of the observer and privileges sight as the basis of rational 
knowledge—has been the preeminent visual model of modernity, as Jay would sug- 
gest. The twentieth century has seen an ‘anti-ocular turn’ and mounting critique 
of ocularcentrism in continental, particularly French, philosophy and scholarship, 
which Jay counters by calling for a plurality of ‘scopic regimes’. According to anti- 
ocularcentrism the privileging of vision as the prime model of perception results in 
an ‘objectification’ of our environment, which exists disconnected from us as a terri- 
tory to be conquered and dominated. 

Cathryn Vasseleu, in her book Textures of Light: Vision and Touch in Irigaray, 
Levinas and Merleau-Ponty (1997), critiques ocularcentrism from a different 
angle by reading the writings of Irigaray, Merleau-Ponty, and Levinas as under- 
mining the dominant characterizations of vision and light in Western philosophy. 
Phenomenology, and Merleau-Ponty’s writings in particular, have gained enormous 
relevance in the field of digital media, since they provide a more appropriate philo- 
sophical and theoretical framework for approaching the forms of embodied interac- 
tion enabled by digital technologies. 

In his chapter on ‘The Spatiality of One’s Own Body and Motility’ in Phenomenology 
of Perception ([1945] 1962), Merleau-Ponty makes a crucial distinction. On the one 
hand there is the body image (schéma corporel) as the visual apprehension of the 
body—the object or content of intentional (noetic) consciousness. On the other hand 
there is the body schema as a flexible, systemic form of distributed agency that 
extends from within the boundaries of the human body to the whole space of embod- 
ied mobility. This is essentially is a prenoetic function. As Shaun Gallagher points 
out, Merleau-Ponty’s distinction between body image and body schema illustrates 
the limitations of the phenomenological method, which, by definition, could not have 
access to the prenoetic role of the body schema (Hansen 2006: 40). 

In his book Bodies in Code, Mark Hansen (2006) uses Merleau-Ponty’s theoreti- 
cal framework, in particular, to analyze the pioneering works of media artist Myron 


Krueger, who countered the ocularcentric paradigm of immersion prevalent in 
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virtual reality with the embodied enaction found in mixed reality environments. In 
his groundbreaking project Videoplace (1974-75), Krueger achieved synchronicity 
between the user’s embodied action and the computer system’s response, allowing 
users to focus on their embodied agency in ‘playing’ the system as an instrument 
rather than understanding the system’s role as interaction partner. As Krueger him- 
self puts it: ‘In the ultimate interface, input should come from our voices and bodies 
and output should be directed at all our senses. Since we will also interact with each 
other through computers. The ultimate interface should also be judged by how well 
it helps us to relate to each other’ (quoted in Hansen 2006: 28). 

While Merleau-Ponty’s theories do not seem directly related to notions of digital 
light, they are in fact crucial to the role that light plays in projected digital environ- 
ments and the way we ‘interface’ with screens. Digital media environments com- 
monly use vision systems that ‘read’ the participant’s silhouette to drive the type of 
embodied interaction pioneered by Krueger. Both Ernesto Klar’s Relational Lights 
(2010) and Scott Snibbe’s Screen Series (2002-2003) are examples of embodied inter- 
actions that are enabled by and question configurations of light. Relational Lights 
uses light, sound, haze, and a custom-software system to produce a three-dimen- 
sional light-space that participants manipulate with their presence and movements. 
Lines projected through the haze onto the floor of the gallery create light-spaces or 
disembodied enclosures within the haze that can be ‘entered’ by people. As partici- 
pants move around in the projected spaces, the lines adapt to the movements of the 
people occupying them, enabling a relational, collective shaping and expression of 
space. Light literally enables the creation of the space and the software-driven pro- 
cessing of people’s movement in it. 

In a different way, Scott Snibbe’s Screen Series also questions the qualities of 
space and light, temporality and movement, and reconfigures conventions of projec- 
tion and perception. Consisting of the works Shadow (2002), Impression, Depletion, 
Compliant, Concentration and Shy (all 2003), the Screen Series questions the cin- 
ematic status of the screen as a mere surface for image projection and turns it into 
a (re)active player in the representational game. Rather than simply being rep- 
resented on the screen, the viewer’s shadow is recorded and played back by the 
screen (Shadow). It changes the screen’s rectangular outline (Compliant), erases it 
or ‘paints’ across it (Depletion, Impression). Screen Series explores relationships 
between bodies, light and shadow by reconfiguring cinematic conventions and 
allowing participants to experience the nature of the image in new ways: while re- 
presentation is still inextricably bound to the process of recording, projecting and 


doubling, every aspect of the representational process becomes a reconfigurable, 
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seemingly active entity. However, Snibbe’s projects are not necessarily cinematic 
in the original sense but ultimately a subtle manipulation of light. The (deceptive) 
nature of the surface and the reflection play a central role in the project, which 
captures complex relationships between reality and representation, our bodies and 
their shadows (as imprint and trace of physical presence), the self and the other. 
In both Klar’s and Snibbe’s pieces disembodied information about our bodies, or, 
applying Merleau-Ponty’s theories, aspects of the body schema, take a concrete dia- 


grammatic form. 


Screens as Interfaces and Mediations of Digital Light 


The previous outline of relationships between light and representation, and the 
vision-centric vs. embodied interaction with digitally projected or encoded light, can 
be applied towards a closer reading of the role that screens now play as mediators 
of light. In the digital age the role of screens has been reconfigured from that of a 
‘passive’ surface that functions as a ‘projector’ of information to one that seemingly 
‘reads’ information about its viewer or environment or even ‘reacts’. The properties 
of light are at the core of many of these reconfigurations. 

The notion of the screen as ‘interface’—an in-between or surface forming a com- 
mon boundary of two bodies, spaces, phases—requires taking a closer look at what 
exactly is being ‘interfaced’. In ‘What Is Interface Aesthetics, and What Could It 
Be (Not)?’, Florian Cramer (2011) distinguishes between the following interfaces: 
hardware to hardware; hardware to software; software to hardware; software to 
software; humans to hardware; humans to software. The following exploration of 
screens, as surfaces on which different forms of information processing meet, will 
predominantly focus on interfacing between software and hardware, humans and 
hardware, and humans to software (= user interfaces). As Cramer has pointed out, 
a substantial amount of research in the area of interface focuses on interface aes- 
thetics as an aesthetics of systems, rather than the aesthetics of interfacing as the 
(social) practices of interaction. 

In ‘Elements of Screenology: Toward an Archaeology of the Screen’, Erkki 
Huhtamo (2004)—who has formulated ‘screenology’ as a specific branch within 
media studies focusing on screens as information surfaces—argues that screens 
should not only be researched as designed artefacts, but also with regard to their 
uses, their intermedial relations with other cultural forms, and the discourses that 


have enveloped them in different times and places. 
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Concepts of the visual and embodied interaction with digital light, in combi- 
nation with interface aesthetics and intermedial relations of the screen will form 
the basis for the following discussion of mediations of light as they manifest in the 
screen as window; watchful and reactive screens; the screen as mirror; the screen 
as membrane; and transposing screens. This classification of screens is not meant 
to suggest a definitive taxonomy but rather a framework for thinking about differ- 
ent functionalities enabled by the interfacing of software to hardware or humans to 


software / hardware. 


Screens as Windows 


Edmond Couchot has argued that ‘the electronic screen does not function like a win- 
dow, it does not inscribe onto a wall, it does not carry the look of the inside towards 
the outside; it inserts, on the contrary, the outside in the inside, in a centripetal and 
violent movement, at the place of the spectator. It acts by way of an inserting effect’ 
(Couchot 1988: 80). While Couchot’s argument may largely capture the effect of the 
TV screen, the interfacing digital technologies of screens complicate the picture and 
often seem to operate precisely on the border of the inside/outside. No matter how 
imprecise it may be, the windows metaphor, a basic element of the concept of the 
desktop, may itself have induced the perception of ‘looking out’. 

Artist John Gerrard has created several projects that, through their use of the 
screen itself as a navigation interface, have underscored the role of the screen as that 
of a window into a simulated digital world. His projects Watchful Portrait (2004)—a 
diptych of two seemingly identical, framed images of a woman (Caroline)—and 
Portrait to Smile Once a Year (Mary) (2006) appear to be photo-realistic ‘head shots’ 
of women that upon closer inspection reveal themselves as computer-generated 3D 
portraits. Rather than being fictitious personae, as most computer-generated char- 
acters are, the portraits are based on actual people, referencing the indexical status 
of photographs. While the works allude to the medium of photography, they also sub- 
tly yet radically undermine the fixed temporality of photographs—the ‘freezing’ ofa 
moment in time. Both portraits include temporal components. In Watchful Portrait, 
the eyes of the two Carolines are tracking the position of the sun and the moon, 
respectively, throughout the course of the day. The precise scientific information as 
to the movement of these elements is constantly generated by the software on the 
basis of latitude and longitude data in combination with date and time. The portraits 
are designed to follow these co-ordinates with their eyes at all times. In one por- 


trait, Caroline opens her eyes at dawn and closes them at dusk, tracking the sun’s 
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movement throughout the day. At dusk, the other portrait of Caroline opens her eyes 
and tracks the moon all night. Natural light and its cycles are at the core of the proj- 
ect yet manifest through a software-driven expression of a portrait. In the case of 
Mary, temporality is stretched further: the portrait will smile once a year (at a time 
determined by its subject or owner), creating a precious, fleeting moment of sentient 
expression. 

The perception of the screen as mirror is created by the fact that viewers can 
turn the framed screen, on which the images appear, on a central pivot point and 
look around and behind the depicted subject. The effect is made possible through 
gaming technology—usually used for the construction of virtual worlds and their 
inhabitants—which Gerrard appropriates for his portraits to make them ‘navigable’ 
in real time. In Gerrard’s works it is the ‘reading’ of the screen’s position that in 
turn determines the perspective of the world on view. The seemingly simple replace- 
ment of conventional navigation devices such as mouse, joystick, or keyboard by the 
screen itself becomes a powerful mechanism for seemingly detaching the screen 


from the scene viewed through its window. 


Watchful and Reactive Screens 


A radically different perception of the screen is produced in projects in which 
screens seemingly ‘watch’ their viewers or react to them. In Rafael Lozano-Hemmer’s 
Surface Tension (1993), the image of a giant human eye precisely follows the viewer’s 
movements. While the motif of turning the gaze back onto the viewer has a long tra- 
dition—Tony Oursler’s projected eyeballs being an obvious precedent among more 
recent works—the software-driven process of Lozano-Hemmer’s work builds a much 
closer connection to computerized surveillance mechanisms and techniques. The 
piece was inspired by the deployment of camera-guided ‘intelligent bombs’ during 
the First Gulf War. The encoding of light in this project is common to many digital 
works: the viewer’s position is read through a vision system and then encoded to 
allow the ‘digital eye’ to follow it. Despite the fact that the content of the project is 
deeply ‘ocularcentric’, the embodied enaction of the observed viewer is key. It is a 
combination of encoded light and movement rather than an actual act of seeing that 
drives the system. 

SVEN—Surveillance Video Entertainment Network (2006-), by Amy Alexander, 
Wojciech Kosma and Vincent Rabaud, with Jesse Gilbert and Nikhil Rasiwasia, 
approaches surveillance from another angle, highlighting its inherent connection to 


entertainment. Fusing the threatening aspects of being watched with the pleasurable 
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thrill of being the focus of attention, SVEN uses a custom computer vision system to 
observe people and assess characteristics that might suggest a potential rock star. 
By default the project takes the form of monitors showing a closed-circuit view of 
the space in which the piece is installed. The vision system ‘watches’ passers-by, 
comparing and matching their movement and body language to that of rock stars in 
music videos. When a person matching the system’s profile of a rock star is detected, 
a real-time video processing application generates visuals and audio reminiscent of 
music videos, incorporating the ‘profiled’ person. These clips, along with the one 
showing the matching rock star, interrupt the closed-circuit feed of the space and 
play on the monitors. SVEN’s tongue-in cheek surveillance illuminates concerns 
about profiling and automated systems as it shifts the focus from being watched to 
how the watching is done. The project explicates concerns about surveillance and 
computer systems as it challenges the assumptions of software-based systems for 
classifying individuals. Once again computer vision drives what is projected via the 
screen, turning it into an interface between human-hardware-software through the 
encoding of light. 

The watchful and reactive screen using facial detection software will, presum- 
ably, become increasingly part of both smart phones and urban landscapes when it 
comes to personalized marketing. The Manhattan-based company Immersive Labs 
developed software that profiles the characteristics (such as gender, age) of people 
on the street, in order to display ads likely to attract them on the digital billboards 
they are passing. SceneTap, an app for smart phones originally launched in Chicago 
and Austin, uses cameras with facial detection software to analyze bar scenes 
(http://scenetap.com/). Without identifying individuals, it assesses characteristics of 
the crowd, such as average age and the ratio of men to women, and, through the app, 
assists bar-hoppers in making decisions about where to go. 

Both facial detection, which does not identify specific individuals, and facial 
recognition, which does, involve analyses in which light plays a crucial role. In the 
abstract for their 2004 article, ‘Appearance-Based Face Recognition and Light- 
Fields’, Ralph Gross, Iain Matthews and Simon Baker from The Robotics Institute at 


Carnegie Mellon University outline their research as following: 


Arguably the most important decision to be made when developing an 
object recognition algorithm is selecting the scene measurements or 
features on which to base the algorithm. In appearance-based object 
recognition the features are chosen to be the pixel intensity values 


in an image of the object. These pixel intensities correspond directly 
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to the radiance of light emitted from the object along certain rays 
in space. The set of all such radiance values over all possible rays is 
known as the plenoptic function or light-field. In this paper we develop 
a theory of appearance-based object recognition from light-fields. 
This theory leads directly to an algorithm for face recognition across 
pose that uses as many images of the face as are available, from one 
upwards. (2004: 1) 


The research is based upon the premise that the plenoptic function or light-field 
specifies the radiance of light along all rays in a scene, and that the light-field of an 
object therefore is the set of all possible features that could be used by an object 
recognition algorithm based on appearance. Since an image is simply made up of 
a subset of measurements from the light-field, Gross, Matthews and Baker reframe 
the already answered question, ‘what is the set of images of an object under all pos- 
sible illumination conditions?’ (2004: 2), and ask the same question about the set of 
all light-fields of an object. The authors draw the conclusion that two objects can 
almost always be distinguished from their light-fields if they have different shapes 
(under arbitrary illumination conditions, two objects that have the same shape can- 
not be distinguished). 

A The New York Times article from 2011 suggested, the average person embed- 
ded in a ‘high-tech’ society will encounter these analyses through the mediation of 
screens that will expose them to a personalized ad or a breakdown of the demograph- 
ics of a bar. The seemingly watchful or reactive screen in these scenarios is a light- 
based distribution device of a light-based algorithmic process (facial detection)—a 


software-hardware-human interface involving several forms of mediation of light. 


Screens as Mirrors 


Yet another, more obviously ‘embodied’ relationship to light unfolds in the use of the 
screen as a mirror. The metaphor of the screen that ‘reflects’, distorts, or remediates 
the viewer has been continuously explored in digital media art. Liquid Views (1993) 
by German artists Monika Fleischmann, Wolfgang Strauss and Christian-A. Bohn, 
for example, recreates a virtual pool of water in the form of a screen embedded ina 
pedestal. Bending over the pedestal, viewers see their reflection on the monitor and 
their touch of the screen produces wave-shaped forms, created by an algorithm, that 
distort the image. Liquid Views both translates the embodied, corporeal experience 
of the reflection into the virtual realm and at the same time unveils the function 


of the interface as a technological device that translates the viewer’s image into a 
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virtual space of reflections. While light may still be crucial in reading the viewer’s 
image, the specular reflection of light is completely simulated. 

While most of the projects discussed here are relying on the traditional com- 
puter monitor / display, a screen displaying digitally driven information can obvi- 
ously take very different materialities. Artist Danny Rozin has created a series of 
‘software mirrors’ and ‘mechanical mirrors’ that play with remediations of mirrors 
and replace the screen with mechanically driven materialities. While this is not nec- 
essarily the intent of Rozin’s projects, they effectively illustrate that the screen is 
just one of many possible interfaces to simulate the effect of a mirror through the 
encoding of light. In Snow Mirror (2006)—from the software mirrors series—the 
image of the viewer, read through a video camera, is re-created through a projection 
of white snow flakes that accumulate in areas of the image that are brighter. Light 
again is the basis for encoding the simulated ‘reflection’. In Weave Mirror (2007), 
the mirror image of the viewer stepping in front of the camera is (re)created by the 
movements of 768 motorized and laminated C-shaped prints that form the screen: a 
smoky, greyscale picture of viewers comes into focus through a gradual rotation in 
greyscale value on each C-ring. Circuitry and wiring are visible behind the picture 
plane of the ‘screen’ and the portrait assembles itself to the sound of the motors. 
Reminiscent of a homespun basket, Weave Mirror mixes textile design and new 
media, evoking the connection between the punched card-controlled Jacquard loom 
and the history of computing. 

Whether they themselves are made of digital materials or not, the screens per- 
forming the function of a mirror in the display of digital information achieve a form 
of embodied synchronicity between viewers and their representations. While light 
is still crucial in the vision system’s processing of the mirrored person’s image, the 
display of the image is not necessarily bound to the reflection of light as in the case 


of Rozin’s Weave Mirror. 


Screen as Membrane 


Screens also frequently fulfil the function of a membrane where the display of infor- 
mation and input from the viewer meet. One could claim that any touch screen cre- 
ates the perception of the screen as a membrane or reactive surface; in the case of 
the touch screen, however, the screen’s function as a form of membrane is not con- 
nected to input dependent on light. 

An example of a relatively early project in which the information displayed on 


the screen seemingly reacted to viewers input, in this case their eye movement, was 
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Art+Com’s Zerseher / De-Viewer (1992). De-Viewer first appears to be a reproduction 
of a traditional oil painting, Giovanni Francesco Caroto’s Boy with a Child-Drawing 
in His Hand (ca. 1515), the first documented child drawing in art history. However, 
the painting slowly starts to blur and disintegrate in exactly the spots where the 
viewer’s gaze meets the work. Whatever is looked at slowly deforms under each indi- 
vidual observer’s eyes, so that the work never repeats itself. If nobody is interacting 
with the painting for 30 seconds, the image is reset to its original condition. 

The reactive effect is created through an eye-tracking system that is analyzing 
the spectator’s gaze and allowing for the exact calculation of co-ordinates on the 
canvas, which are then sent to a graphics program that in turn distorts the picture. 
Commercially available eye tracking systems are frequently based on video images 
of the eye. The trackers capture reflections of infrared light from both the cornea 
and the retina and thereby employ the process of ‘seeing’ to turn vision upon itself 
and track it. While De-Viewer appears to be inherently ‘ocularcentric’ it also expands 
the natural limits of vision by giving it destructive material properties. The project 


allows for embodied action that transcends the capabilities of the sense of vision. 


Transposing Screens 


At least since the beginnings of closed-circuit video art, experimentation with live 
reconfigurations of space that challenge embodied spatial and temporal perception 
has become a wide area of artistic practice. Video artist Dan Graham, in particu- 
lar, created a number of installations that challenged the perception of space, or 
assigned the audience the double role of performer / viewer by making them experi- 
ence themselves with a temporal delay; see for example, Time Delay Room (1974) or 
Opposing Mirrors and Video Monitors on Time Delay (1974). The possibilities of digi- 
tal technologies have taken these temporal and spatial explorations and transposi- 
tions of views to a new level. 

Kazuhiko Hachiya’s Inter Dis-Communication Machine (1993), for example, is a 
communication system that transposes visual and sensual experiences. Used by two 
people wearing head-mounted displays, the ‘machine’ projects one wearer’s sight 
and sound perception of the environment into the other one’s display, thus confusing 
the borders between the identities of ‘you’ and ‘me’. The Inter Dis-Communication 
Machine allows its wearers to seemingly enter each other’s body and perception 
without being able to influence it. In a very different way than projects employing 
computer vision systems, Inter Dis-Communication Machine deliberately breaks with 


the synchronicity between the user’s embodied action and the computer system’s 
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response while still emphasizing the vision-centric paradigm of immersion. Inter 
Dis-Communication Machine creates an out-of-body experience disrupting its users’ 
body schema—the boundaries of their bodies and their space of embodied mobility— 
in order to make them aware of the body schema’s function. 

As the above examples show, digital technologies have substantially expanded 
the ways in which light is mediated. In any form of digital ‘imaging’ the encoding 
of light can occur during a project’s creation, editing, and display via a digital sys- 
tem (be it a screen or projection). Screens are just one manifestation of interfaces 
that allow for mediations of light; yet they already involve enormous complexities. 
In many cases screens technically are just display mechanisms for a software-driven 
process (and mediation of light); in other cases (as in John Gerrard’s works) they 
become an interface for navigation. 

On the basis of Huhtamo’s research one could argue that the use of screens, the 
intermedial relations they enable, and the discourses surrounding them defy merely 
technical explanations of the role that they play as information surfaces and media- 
tors of light. The role of the screen as a watchful and reactive device or as a window, 
mirror, or membrane is not only assigned by technicalities but by perception, which 
is physically and culturally shaped. 

Digital technologies can be said to expand the role that light plays in our engage- 
ment with the world: light is not only a basic principle of vision—tied to the ocu- 
larcentric paradigm—but also an essential element in mediating embodied action. 
The ‘reading’ and analysis of light and shadow usually is at the basis of computer 
systems that support the role of a screen as reactive or watchful or mirror. The com- 
plexities of the digital mediation of light expand the functionality of the screen from 
that of a display mechanism for pre-processed information to a surface on which 


multiple forms of information processing—from vision to motion—intersect. 
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CHAPTER 10 


View in Half or Varying Light: 
Joel Zika’s Neo-Baroque Aesthetics 


Darren Tofts 


It is an image that has been glimpsed before. Many times before. A solitary figure 
is poised on a threshold. On the precipice of a cliff, the edge of a cave or the ver- 
tigo of doorways within doorways in confined domestic spaces. Or more ambiva- 
lent liminal spaces that are anticipatory, between states, entrances to unfamiliar 
interior or exterior space. This moment of anticipation may be one of enchantment, 
such as Alice’s fall into Wonderland down the rabbit hole or melding with it through 
an impossible, membranous mirror. When Dante and Virgil ascend from Hell in the 
Inferno they are in abeyance between condemnation and purification, faced with the 
tantalizing glimpse of a paradise they have yet to achieve. The domestic interiors of 
Jan Vermeer and Pieter de Hooch create recursive, manifold spaces in which open 
doors, archways, latticed windows and mirrors suggest deeper recesses within, 
whereby an opening becomes the portal to yet another space. In Bram Stoker’s 
Dracula (1897) Jonathan Harker is beckoned ominously to cross the threshold into 
his host’s castle ‘freely and of your own will!’ This movement across physical and 
metaphysical thresholds is a decisive figure in that quintessentially American genre 
of the Western. Arguably its most totemic expression is encountered in John Ford’s 
The Searchers (1956). In its iconic opening sequence the silhouette of a woman 


opens a cabin door from the darkness of an unseen interior. The sweeping vista of 
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the Texan frontier onto which she gazes is the glaring and untamed wild nature 
that pitilessly illuminates her for our gaze in the counter shot. Her disappointed 
countenance, we quickly realise, suggests yet another frustrated hope of an arrival 
yet to come. A similar frontier drama is encountered in Neo’s revelatory moment of 
déja vu in The Matrix (1999): ‘A black cat went past us, and then another that looked 
just like it’. Neo’s seeing double is a sign that the invisible seams of the panopti- 
cal Construct have momentarily frayed, its omnipresence revealed as the simulacral 
order of things is subtly modified in response to his presence. All of these threshold 
moments are forms of estrangement, defamiliarizing epiphanies in which someone 
sees things differently, perhaps as if for the first time. 

This poetic of someone watching and being watched, the undifferentiated sur- 
veillance of ambiguous spaces, of spaces within spaces, underpins the aesthetics of 
Australian media artist Joel Zika. Specifically, Zika’s work engages with troubling 
perspectives of indeterminate and often vestigial environments that are viewed in 
half or varying light; a subtle tonality of luminance and shade that evokes an immi- 
nent crossing of a threshold (see http://joelzika.com). Though not just any thresh- 
old, one with consequences, an ambivalent passage described by the artist as ‘an 
entrance into nowhere’ (Zika 2009: 43). Joel Zika is one of a number of contempo- 
rary Australian artists discussed by Simon Gregg in his 2011 book New Romantics: 
Darkness and Light in Australian Art. Gregg is specifically interested in the ways in 
which the richly nuanced binary of darkness and light is played out in contemporary 
Australian art. Gregg is concerned with the ways in which artists as varied as Bill 
Henson, Jane Burton and Kathryn Ryan have engaged with Romantic notions of the 
sublime in their work, or have attempted to go beyond it to explore post-Romantic 
tropes such as the revival of the grotesque. In Zika’s work the manipulation of digi- 
tal light is also treated quite differently from the pastoral sublime yearned for in 
the work of the other artists identified by Gregg. Zika’s work—specifically the two 
series At Night (2005) and Night and Morning (2008)—evocatively and disturbingly 
suggests a resurgence of the Baroque. 

Zika’s work is not Baroque in any classical sense of the word, as it is understood 
within art history. But it revives aspects of its theatricality, artifice and taste for 
staged trompe l'oeil set pieces in order to create robust, immersive visual spaces of 
unnerving enchantment, mystery and estrangement. As a media artist Zika is par- 
ticularly known for his ongoing interest in the history of the American sideshow 
attraction of the ‘dark ride’. With its origins in the deeper pre-moving image history 
of what writer Richard Abel described as a ‘cinema of attractions’, such as ghost 


trains (Abel 1994: 16), the dark ride represents for Zika a stage in the history of 
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media technology’s implicit fascination 
with the unseen, the otherworldly and its 
manifestation through sound and light. 
Contemporary media artists have 
demonstrated an especially keen interest 
in the connections between the spectral 
and the sideshow attraction, particularly 
as a point of intimate, interactive encoun- 
ter. Canadian artist Catherine Richards’ 
installation I was scared to death; I could 


have died of joy (2000), for instance, con- 


structs an enclosed world that evoca- 


Figure 1. Catherine Richards, | was scared to death; | could have 
tively blends the curiosity cabinet and died of joy, 2000. Photo: Mitch Lenet. Courtesy of the artist. 


the freak show. 

Two glass canisters resembling nineteenth century specimen jars are set on 
stainless steel tables in a dimly lit room. The jars, in actuality gas-filled vacuum 
tubes, contain life-like glass models of a cranium and a spinal cord. As the visitor 
approaches them the light recedes even further until the room is completely dark. 
As if sensing this as a kind of bizarre curtain call in a Victorian pathology museum, 
each object glows in a responsive fluorescence. And it responds to your presence. 
Katherine Hayles has evocatively described this experience: 

When an ionizing current is passed through the gas, the glass models fluoresce, 
creating a beautifully eerie glow ... [As the ionized gas] arcs towards the visitor’s 
hand, a connection and possibly a communication seems to be established between 
the interiority of the glass nervous system and the cognition of the visitor as she 
meditates on the experience. (2000, 10) 

The play of light and shadow, brightness and darkness in this work has been 
described by Hayles as a kind of allegory of our experience of light itself. ‘Light’, she 
suggests, ‘is the portion of the electromagnetic spectrum visible to humans’ (2000: 
10).1 Her response to this foreshortened dimension of vision is to suggest, therefore, 
that for the most part ‘we see darkly’. The act of seeing darkly is an apt figure for 
thinking about Zika’s fascination with the complementary as well as contradictory 
qualities of digital light. 

As attractions that simulate the crossing of thresholds, the unnerving descent 
into the otherworldly dark rides represent for Zika incipient, premonitory and pow- 
erful examples of immersive, virtual worlds. They are potent admixtures of light 


and shade, uniquely hybrid spaces that combine the mechanics of familiar trolley 
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Figure 2. Joel Zika, Arcade # 1, 2006. Courtesy of the artist. 


rides (such as the rollercoaster), the fabricated construction of the walk-through 
built-environment (haunted house) and the emergent moving image technology of 
the cinema. Zika points out how such attractions were indelibly associated with 
the rapid and pervasive urbanism of early twentieth century America: ‘As people 
became increasingly alienated from their local environs, they sought alternative 
destinations in a variety of forms of escape involving physical as well as imaginary 
sorts of “transport” (Zika 2009: 12). Geographically, the locations of these gothic 
amusement parks were often found in ‘a remote, isolated space’, in the woods, on 
the edge of town, or a patch of wasteland (Zika 2009: 13). Symbolically at a distance, 
as if their horrors could not be countenanced in the streets where people actually 
lived, this of course added to their titillation as attractions. You had to make a com- 
mitment to go to them, a commitment to cross a threshold that assuredly would 
set the hair tingling on the back of your neck. Zika cannily captures this psycho- 
geography of expectation in a number of image-based and installation works. For 
him it represented a key aesthetic and psychological component of the cultural reso- 
nance and meaning of such spaces, since the physical journey to get there, which 
took you to the ‘end of the line’, was also a ‘mythological journey into the unknown’ 
(Zika 2009: 13). 

An iconic work in this respect is Arcade (2006). A stylized ghoul-like visage glows 
eerily from an undifferentiated background surrounded by trees. The silhouette of 
a young girl is seen approaching this unnerving portal, her hand reaching out as 
if feeling her way through the dark. The chilling second panel of the work reveals 
the child on the verge of entering into an abstract space (an image also witnessed 
in Façade of the same year). Frozen as a kind of cipher, a filtered image of light, 


she is forever captured in the intransitive state of entering. She is captured as a 
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x 


Figure 3. Joel Zika, Arcade #2, 2006. Courtesy of the artist. 


tableau vivant beyond recall and, potentially, beyond salvation.? Later installation 
works such as Inferno (2007) and Terrorium (2009) are uncannily devoid of such an 
expectant human presence, revealing perhaps reflexively the indifferent and solemn 
spectacle of a moment that has since passed. All that remains is the glow and envel- 
oping saturation of light. 

Central to the spectral, gothic atmospheres of the dark ride was the interplay 
between light and shade. Indeed, the historian of early cinema Laurent Mannoni 
describes pre-cinematic projection as a ‘deceptive art’, the ‘great art of light and 
shadow’ (2006: 42). The manifestation of the otherworldly as spectra of light has 
been traced back many times to the age of the magic lantern. The cone of light 
channelling through a darkened space bore an uncanny resemblance to steam, the 
psyche of the machine age. But more alarmingly it is also suggestive of breath, a 
psychosomatic aspect of projected light that Zika evocatively captures in his work, 
especially the At Night series (2005). Zika’s interest in the palette of light and shade 
is continuous throughout his work in different media from series comprised of still 
images to immersive installation environments (such as the large scale walkthrough 
‘amusement’ Terrorium). I want to focus particularly on At Night to concentrate the 
ways in which Zika mobilizes the seductive malleability of digital light. A latter- 
day Franz Mesmer, Zika invests the valencies of digital light and shade with the 
animus of character, of psyche and an enveloping sense of presence. His scenes, in 
this sense, can be considered less as planar ‘inkjet’ photomedia images than the 
orchestrated interactions of an ensemble of dramatis personae (mise-en-scène is far 
too static, less animate a term here). This performative vibe may partly be explained 
in terms of the artist’s actual experience of the vestigial and often decaying theme 
parks he visited in the United States in 2007, such as Coney Island’s ‘Spookarama’ 
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Figure 4. Joel Zika, At Night #1, 2005. Courtesy of the artist. 


and the ‘Old Mill’ at Kennywood in Pittsburgh. In this sense the dark ride as a form 
of neo-Baroque representation, captured acutely in the At Night series, is the final, 
decadent flourish of a medium that would be succeeded by the cinema, the residue 
of a once former glory that still retains the power to unnerve and disturb. 

Installed as a panorama of four images, At Night is immediately conspicuous for 
its absence of the human form. This immediately and uncomfortably situates the 
spectator as the focal point of view of the work. The first image reveals the facet of 
what appears to be an art deco façade of a building partially concealed by fog and 
the silhouette of a tree brightly illuminated from behind by an unseen source.* Next 
to ita sharply illuminated portico or entrance (not dissimilar to those of Inferno and 
Terrorium) manages to outshine the strong backlight of the previous image, though 
its saturation is still present in the casting of the winter trees as flattened out sten- 
cils in the foreground. In the third image a vulnerable distance seems to have been 
placed between the viewer and the environment of the previous two. We are some- 
where else and there’s no one else around. An old trolley car sits becalmed, but also 
seems out of place. Why is it here, as if concealed among the trees? The sharp light it 
seems to have absorbed is so clear that it reflects its own sienna hue onto the trees 
that conceal it, staining them a deep blood red. As if this drama of light, reflection 
and tension has been too much to bear, the final image of a tree in leaf brings some 
relief, though its complete lack of placement in any kind of identifiable landscape is 


still cause for concern. 
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The drama of this sequence of four 
modest images lies in its ambivalence. 
It at once resonates with the signs and 
atmosphere of charged expectation, 
as well as the desire to be frightened. 
A rudimentary journey into an uneasy 
heart of darkness guides the eye from 
the approach to the threshold moment of 


anticipation, the culmination of the ride 


and the return to the world of nature 
that is comfortably beyond the fabri- Figure 5. Joel Zika, At Night # 3, 2005. Courtesy of the artist. 
cated terror that brought you here in the 
first place. Certainly my own experience 
of this work in situ came with the uneasy 
intimation that I seemed to be the only 
living thing in the scene (despite the fact 
that there were other people in the gal- 
lery); an uncanny sensation generated 
by the subtle vanishing points of light in 
each image that insinuated that my gaze 
was its solitary locus. But more viscerally 


there is a compelling sense of presence. 


Alain Robbe-Grillet famously described 


Figure 6. Joel Zika, At Night # 4, 2005. Courtesy of the artist. 


the experience of Samuel Beckett’s 

dramatic works in terms of presence, 

the overwhelming sensation that you are ‘on stage’, you are ‘there’ (Robbe-Grillet 
1965: 111). The same is true of At Night. Whether you like it or not, you are not an 
aloof spectator. This sensation of being there has gone by many names in visual and 
dramatic aesthetics, such as identification, agency and empathy. For Heidegger, as 
Robbe-Grillet reminds us, ‘to be there’ is the human condition itself. But it is also the 
psychopathology of what we today understand as immersion in virtual space, from 
installation environments to digital images. Writing in the mid-1950s Beckett him- 
self talked about ‘the power of the text to claw’ (Beckett 1962: 183), a potently tactile 
image that reminds us that imagined or simulated presence is ultimately concerned 
with affect. T. S. Eliot, writing at a time when pre-cinematic projection media were 
familiar and still in use, captured this somatic quality of our engagement with phan- 


toms of light in a decisive line from ‘The Love Song of J. Alfred Prufrock’ (1917): ‘But 
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as if a magic lantern threw the nerves in patterns on a screen’. This is the uncom- 
fortable, non-negotiable presence Zika invokes in his manipulation of digital light. 

At Night is a sequence of images unavoidably experienced in half or varying light. 
The experience of ‘sound-in-space’, Erik Davis suggests, ‘does more than just shape 
a sound world; it also transports the listener’ (Davis 2005: 71). The same can be said 
of the activity of vision and the manifold, tonal nuances of light. Light is the elemen- 
tal force that transports the viewer as a somatic projection into the work. In his 
splendid book on Led Zeppelin’s ‘runes’ album (colloquially known as Led Zeppelin 
IV), Davis picks up on Jimmy Page’s oft-quoted formula of ‘light and shade’ as a way 
of describing the band’s orchestration of the elemental polarities of lyricism and 
hard rock, lightness and aggression, brightness and darkness, pop and cock rock. 
The subtle modulation of light in the At Night series not only transports the viewer, 
but also creates the vicarious sensation of being in the presence of a certain kind of 
energy. In his famous 1975 Crawdaddy interview with William S. Burroughs, Jimmy 
Page spoke of the rock concert as an unleashing and mobilisation of energy through 
sound. The guitarist is a kind of magus, though one with a conscience, channelling 
sound in ways that blur the thin metaphysics between good and evil, light and shade 
(Burroughs 1975).4 The same can be said of artisans of digital light such as Zika. It’s 
important to note that the phrase ‘view in half or varying light’ that gives this essay 
its title and working metaphor, is the name of the painting by Barrington Colby that 
lines the left hand side of the inner gatefold of Led Zeppelin IV. Without question an 
instruction, the phrase was one of many cabbalistic references associated with all 
aspects of the album (from its liner notes, cover art and runic title to its lyrics). Once 
deciphered, it was understood as a direction on how to hold the Colby illustration at 
right angles against a mirror. Alchemy then ensues. The doubling of the image on 
itself creates the unmistakeable likeness of a monstrous dog, the eponymous black 
dog of the album’s first track. Ambivalence implies in one of its connotations the 
possibility of an either/or dichotomy. So what may be a hound from hell may just be 
a misprision, the kind of wishful thinking that prompted a generation to decipher 
back-masked messages or tantalizing lead-out grooves on Beatles albums for yet 
another ‘clue for you all’ to do with the untimely death of Paul McCartney. 

But ambivalence is also either and or, the uneasy co-presence of a thing and its 
opposite. The ambivalence of the admonition to ‘view in half or varying light’ can 
then be re-read as half and varying light, good and evil, expectation and dread, after- 
math and anticipation. Let’s not forget that in an age when television didn’t exist and 
the cinema was in its infancy, the sole objective of the dark ride was to scare the 


wits out of you. As Zika points out in his study of the dark ride, these meandering, 
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Figure 8. Joel Zika, Night and Morning, 2008. Courtesy of the artist. 


clunky ‘pretzel’ contraptions were holistic mixed media and multi-sensory experi- 
ences. The visceral nature of movement, the force of air through the hair, sound, 
smell and tactility meant that for a short time out-of-time, you can experience dread. 
In his photomedia work Zika exploits the synaesthesia of digital light to attune the 
senses to other channels of information to create this sense of discomfiture. 

The iconic image of a death’s head in Façade or a fourteenth century Taddeo 
di Bartolo-inspired Lucifer inviting you into the depths of his bowels in Night and 
Morning are figurative encounters with familiar images of dread. Their familiarity 
presumes a palatable distance, a kind of narcosis and a measure of safe horror that 
representations have always provided (what Aristotle understood as catharsis). 

But At Night doesn’t allow such a release valve. What happened at the moment 
when that anonymous figure went beyond the threshold in Facade? Or more dramati- 
cally, when the young girl in Arcade took what would seem to have been a fatal step 
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into the unknown? There is no hokey counter shot in either work that allows us to 
share their sense of horror. Their terror is not represented, since it is beyond repre- 
sentation. There is no silent witness for whom it can become an image. As with that 
other famous literary image of a crossed threshold into the Marabar caves in E. M. 
Forster’s A Passage to India (1924), it is an aporia, an irresolvable contradiction of 
which the truth can never be known. All we are left with as readers, as viewers, is 
the menace of the unknown, the signs of absent presence. This oppressive intuition 
that something unspeakable has happened prior to our arrival haunts the At Night 
series like a malevolent miasma. In this sense the images bristle, like so much static 
electricity, with a profound sense of aftermath that invites an act of speculation. 
This is the ‘forensic aesthetic’ described by art critic Ralf Rugoff, the realisation 
that we are privy to a place where ‘something happened’ (1997: 62). The legendary 
American hard-boiled crime photographer Weegee (aka Arthur Fellig) was always 
the first to arrive at such scenes, photographing the body before it had time to cool. 
But rather than simply documenting that which has happened, he was concerned 
with the aesthetics of the image, with ‘the way they look’. And more pertinently, 
he was interested in the ways in which an individual image ‘fits within a series’ 
(Wollen 1997: 31). 

Armed with the forensic aesthetic as a mode of textual analysis, At Night dramat- 
ically changes in terms of our familiarity with Zika’s dark ride works as a whole, as 
a series. Our perspective shifts from an anxious, solitary presence to a speculative 
gaze of urgent inquiry. No longer simply there to experience the ambience, we feel 
compelled to piece together a series of clues that bespeak a narrative that remains 
‘largely invisible to the eye’ (Rugoff 1997: 62), though a narrative tempered by what 
Peter Wollen has evocatively described as the ‘vectors of melancholy’ (1997, 23). 
From cinéma vérité style glimpses of sideshow attractions, the images now resemble 


a set of crime scene photographs without a body. The work of mourning has begun. 


References 


Abel, Richard. 1994. The Ciné Goes to Town: French Cinema 1896-1914. Berkeley: 
University of California Press. 


Beckett, Samuel. 1962. ‘Letter to Alan Schneider, 21 June, 1956.’ In The Village 
Voice Reader, edited by Edwin Fancher and Daniel Wolf, 166-69. New York: 
Doubleday. 


Burroughs, William S. 1975. ‘Rock Magic: Jimmy Page, Led Zeppelin, and a search 
for the elusive Stairway to Heaven.’ Crawdaddy, June. 


View in Half or Varying Light: Joel Zika’s Neo-Baroque Aesthetics 203 


Davis, Erik. 2005. Led Zeppelin IV. New York: Continuum. 


Hayles, N. Katherine. 2000. ‘Supersensual Chaos and Catherine Richards.’ In 
Excitable Tissues. Ottawa: The Ottawa Art Gallery. Exhibition catalogue. 


Mannoni, Laurent. 2006. ‘The Art of Deception.’ In Eyes, Lies and Illusions, 41-52. 
Melbourne: Australian Centre for the Moving Image. Exhibition catalogue. 


Robbe-Grillet, Alain. 1965. For a New Novel: Essays on Fiction. New York: Grove 
Press. 


Rugoff, Ralph. 1997. ‘More than Meets the Eye.’ In Scene of the Crime, 59-108. 
Cambridge, Mass.: MIT Press. 


Wollen, Peter. 1997. ‘Vectors of Melancholy.’ In Scene of the Crime, 23-36. 
Cambridge, Mass.: MIT Press. 


Zika, Joel. 2009. ‘The Dark Ride: The Translation of Cinema into Spatial 
Experience.’ MFA diss., Monash University. 


Notes 


1. This work was first installed at the Galerie d’art d’Ottawa in 2000. It was also exhibited 
at the 2004 Sydney Biennale. For images of this work see www.catherinerichards.ca/ 
html/brain.htm. 


2. Arcade is very much a site-specific work. Installed in a public exhibition space in 
Manchester Lane, Melbourne, the work references the infamous murder of 12 year old 
schoolgirl Alma Tirtschke in 1921, whose body was found in a nearby alleyway. 


3. The construction of the image is suggestive of Zika’s approach to the treatment of 
light as a malleable material that can be translated through and between media. 
The art deco façade is modelled on the ‘Laff in the Dark’ ride at Lake Compounce in 
Bristol, Connecticut. Zika photographed it in situ in 2007. As with most vernacular 
photography of this kind it is intended as a testimonial image to his actual presence 
at the site. The photograph then formed the source material for the compositing of 
a digital image capable of simulating a different kind of presence for his viewers 
when the work was exhibited at the Spacement Gallery in Melbourne in 2006. In both 
instances, someone in these images is ‘there’. 


4. In the introduction to the Crawdaddy interview Burroughs reminds his readers that 
the origin of all the arts is ‘magical in purpose’ and is concerned with the ‘evocation 
and control of spiritual forces’. For such magic to succeed, Burroughs warns, ‘it must 
tap the sources of magical energy, and this can be dangerous’. With the notion of 
responsibility to his audience in mind, Page observes, somewhat ominously, that ‘we 
don’t want to release anything we can’t handle’ (Burroughs, 1975). 


CHAPTER 11 


The Panopticon is Leaking 


Jon Ippolito 


This chapter traces the historical roots of light as both a metaphor for knowledge 
and a means of control, and questions the relevance of this legacy for the era of 
networked dissidents such as Anonymous and WikiLeaks. As I will argue, the anal- 
ogy between light and information is hampering our understanding of politics in 
the Internet age, even if such outdated metaphors are proving stubbornly difficult 
to discard. In the Internet age, a previously dominant broadcast paradigm has lost 
ground to a network paradigm, and it is within this context that I’m going to propose 
and analyse updated metaphors suggested by recent art and activism.! 

The concept of light has no independent meaning in contemporary physics; light 
is simply a particular swath of wavelengths on the electromagnetic spectrum, nes- 
tled snugly between infrared and ultraviolet radiation. So-called ‘digital light’ is 
an even more arbitrary category, since the light that shines from our laptops and 
smartphones starts out as voltage differentials on a hard drive. If the word ‘light’ is 
to have any scientific significance, then it is distinguished not by any of its intrinsic 
physical properties, but by the biological fact that humans can see it. 

Of course, Euroethnic culture has inherited from Enlightenment metaphysics a 
deep-seated sense that light is the emblem of truth, rather than a biological con- 
tingency. As René Descartes stated, ‘Whatever is revealed to me by the natural 
light ... cannot in any way be open to doubt’ ([1641] 1904: 38). Even today, despite 
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the fundamental transformations in physics over this period, our language betrays 
our debt to Descartes’ legacy. Web designers aim to attract a lot of ‘eyeballs’ to a 
Website, to maximize its ‘visibility’. ‘Watchdogs’ press governments to enact ‘sun- 
shine’ laws to guarantee ‘transparency’. The closer we look at our dependence 
on the metaphor of light as truth, the more we will see a political expediency hid- 
den beneath it. 

I once ate at an Asian restaurant where I received a fortune cookie that con- 
tained a rather Orwellian message: ‘There is something seeing, and there is some- 
thing being seen’. Sipping my green tea, I imagined the woman who inserted this 
fortune into the cookie, working in a nondescript factory under the watchful eyes of 
an overzealous night shift supervisor. However, those human eyes are increasingly 
being replaced, according to the relentless logic of a globalized economy, by the 
more efficient digital eyes of hidden security cameras. So the fortune cookie really 
should have said: ‘There is something seeing, but it is not being seen’. It is precisely 
this asymmetry that makes our unexamined metaphors of light so dangerous, and 
helps the powerful instil fear in the powerless. 

The security camera watching the fortune cookie lady is of course a modern ver- 
sion of the Panopticon, Jeremy Bentham’s notorious 1791 design for a maximum- 
security prison. Bentham’s plans called for a central guard tower whose windows 
look out onto an annular building divided into prison cells. Impenetrable walls divide 
the inmates from each other and each cell is designed with a window facing back at 
the tower, as well as another window on the opposite wall facing the outside world. 
The function of this second window is not to give inmates a room with a view, but 
to backlight them against the light of day, making them all the more visible to the 
inspectors in the central tower. 

The inspectors employ Venetian blinds to ensure that, at any given moment, 
inmates cannot tell whether they are being watched. While Bentham had in mind 
a limited application of his design to prison architecture, the metaphor of the 
Panopticon has been appropriated widely by scholars and has come to represent 
structures of power more broadly. Michel Foucault famously called the Panopticon 
‘the diagram of a mechanism of power reduced to its ideal form’, analysing its influ- 
ence in the nineteenth and twentieth centuries on the architecture of hospitals, 
workshops and schools. Foucault keenly realised the applicability and effects that 
the Panopticon model could have in any situation where ‘one is dealing with a mul- 
tiplicity of individuals on whom a task or a particular form of behaviour must be 
imposed’ (Foucault 1977: 205). 
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Figure 1. ‘Plan of the Panopticon’, 1843 (originally 1791). From The 


Figure 1 shows a layout of Bentham’s 
Panopticon; however it could easily serve 
as an infographic of the IT department’s 
recommendations for the $10 million 
media centre built on my university’s 
` campus in 2013. These recommendations 
included a newly proposed ‘information 
security policy’ that would require fac- 
ulty to observe a ‘clean desk’ policy, pro- 
vide an inventory of everything on our 
laptops, and to ask permission to take 
them home at night. The insouciance 
with which we yoke the incompatible 
words ‘new media’ and ‘centre’ into the 
same phrase shows how persistent pan- 
optic attitudes remain. 

Of course, ‘digital’ light is not a single 
sun in the sky shining down to provide 


chiaroscuro for Renaissance painters or 


Works of Jeremy Bentham. Vol. IV, 172-3. Image courtesy Wikimedia backlighting for Panopticon wardens; 


Commons, accessed 2 March, 2014. http://en.wikipedia.org/wiki/ 
File:Panopticon.jpg. 


it is a dispersed array of handycams, 
Webcams, phonecams, dronecams and 
software agents like face-recognition 
bots. Yet in spite of digital light being more dispersed it is also less trustworthy, 
as evidenced by the ease of manipulating a digital image through Photoshop. If 
Descartes were alive today, he would have to write: ‘Whatever is revealed to me by 
unnatural light ... cannot in any way be trusted’. 

If that is true, then why is digital light still ascribed the power to control that 
Descartes originally attributed to natural light? As we move out of a broadcast par- 
adigm and into a network paradigm, those metaphors of vision and its attendant 
radial model of information gathering and dissemination are increasingly irrelevant 
for a world lit not by a single light source in the heavens, but by a billion strands of 
interlinked fibre optics and Wifi networks. The metaphor of light endures because 
it slides conveniently into a metaphysics of asymmetry, which happens to be a very 
convenient metaphysics for government officials, university administrators, CEOs, 


or anyone who wants to remain in the position of power that he or she occupies. 
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The year 2007 saw the launch of what would become the most far-reaching 
example to date of the powerful using surveillance to maintain control. This secret 
initiative tapped into an extraordinary swath of live and stored Internet commu- 
nication, from email to live audio and videochat, to cloud data stored by compa- 
nies like Google and Apple. It collected private information of Internet and phone 
users across the world, including leaders of allied countries, and quickly became 
the primary source of raw intelligence used for the United States’ National Security 
Agency. In 2013 NSA contractor and whistleblower Edward Snowden revealed this 
clandestine program to the world, prompting headlines about this ‘shadowy’ spyplot 
completely ‘hidden from public view’. And it was not just journalists who resorted 
to the metaphor of knowledge as light; the program’s very codename, PRISM, sug- 
gests the ability to filter the ‘white noise’ of signals from data ‘clouds’ into precise 
frequencies that can be individually observed and analysed.’ 

Perhaps presciently, January 2012 saw the ‘empyre’ discussion list? attempt to 
update the nineteenth century concept of the Panopticon for the twenty-first century. 
The original Panopticon, as its name implies, depended on light as a means of gath- 
ering information. But the means by which light illuminates a perspectival space is 
different from the way that electromagnetic signals—even if travelling through fibre 
optic cables—spread knowledge through networks. Perspectival light is a broadcast 
medium; it radiates. An illuminated object reflects light outward in all directions 
from its position. Conversely, a central eye, by swivelling in different directions, 
can monitor incoming light from every quarter. A guardian of the Panopticon could 
do a quick count of the entire prison population simply by panning his glance 360 
degrees across the rows of cells in the peripheric ring. However there is a natural 
limit to such an illuminated landscape: the horizon. Potential threats from barbar- 
ian invaders would come from the horizon, necessitating lookout towers that could 
see such enemies before they reached their target. Conversely, the horizon is also 
a point of possible escape; thus the Panopticon functioned to control the sight lines 
through which inmates would attempt to flee. 

Now that the backlit horizon of the Panopticon has been replaced with a backlit 
screen, we can no longer always see an invader before it is upon us. Paul Virilio has 
noted that in previous eras military control hinged on keeping the target always 
in sight, but in the future the means of control may not depend on visual contact. 
Reconnaissance can be replaced by a global Search, strikes by Search and Delete, 
invasion by Search and Replace, colonization by Search and Replicate. As the forces 
of ‘offence’ abandon the Panopticon in favour of networked attack, so the defence 


must abandon the medieval tower. In Panoptic space, the height of the lookout tower 
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enables a more comprehensive control of information. In cyberspace, however, con- 
trol is maintained not at a high level but at a low one. The lowest level of computer 
code is the most powerful, which is why data-miners reap profits by tapping into TCP/ 
IP streams and hackers dig deep into code to inject rogue SQL and other malware. 

Information no longer radiates from a point outwards in all directions, but rather 
it follows diverse and sometimes multiple pathways to its destinations. If an inmate 
managed to escape from an ordinary prison, the authorities might suspect a con- 
spiracy and would therefore interrogate the next-door neighbours in the cellblock. 
It was precisely to prevent such collaboration that Bentham designed walls between 
adjacent cells in a way that meant contiguous prisoners could not see each other. 
However today in cyberspace, the tendency of information to suffuse the local neigh- 
bourhood is replaced by a tendency for information to proceed through one or more 
discrete vectors. Governments and corporations now interrogate nodes instead 
of neighbours, hoping to trace the routes that information takes in the Internet 
age. This is why Egyptians organizing online protests in 2011 hid their activities 
using the Tor ‘onion router’, which was designed to bounce network traffic through 
untraceable pathways. Unfortunately, not all netizens are as careful to hide their 
tracks. Bentham’s guards locked inmates behind iron bars, while wardens of today’s 
virtual enclosures have designed fishbowls that victims inhabit voluntarily; they’re 
called Facebook accounts. 

Architect Malkit Shoshan christened the twenty-first century Panopticon 
spawned by the Internet the ‘Netopticon’. A more accurate coinage would have been 
PanNetwork, since light is only an incidental medium for electronic networks. By 
clinging to an optical etymology, the term Netopticon betrays the difficulty we have 
in discarding the metaphor of light as information. In a Netopticon, networks serve 
as simply another conduit for light to bring information from the powerless to the 
powerful; advertisers mine Facebook for example, and the FBI mines Twitter. In a 
truly distributed network, however, information is not a commodity for privileged 
nodes to broadcast or trade privately; information is a property of the system. That’s 
why in its recent spying on phone calls the US government chose to store metadata— 
which reveals the connections among events—in addition to or in place of recording 
the actual calls. It’s also why the government’s claim that this practice is more sani- 
tized doesn’t ring true. Consider this example cited by Princeton computer scientist 


Edward Felten on behalf of the American Civil Liberties Union: 


A young woman calls her gynecologist; then immediately calls 


her mother; then a man who, during the past few months, she had 
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repeatedly spoken to on the telephone after 11pm; followed by a call 
to a family planning center that also offers abortions. A likely storyline 
emerges that would not be as evident by examining the record of a 


single telephone call. (Washington’s Blog 2014) 


This is also why light is such an inadequate metaphor for information in a network.* 
There is no single vantage point from which all these events are visible. The most 
information on the planet sits where light cannot reach it, in the rhizomes and 
mycelia under the soil, in the nervous systems and brains of mice and men, in what 
neurologists call the ‘connectome’. As Joline Blais argues, the Enlightenment was 
about the trees—in Diderot’s words, ‘encompassing each and every branch of human 
knowledge.’ In the twenty-first century, it is about the roots (Blais 2011). 

So what metaphor would be appropriate to describe networked information? 
Marshall McLuhan claimed in The Global Village (1989) that electronic networks are 
based on an acoustic, rather than an optical, world-view in that they permit over- 
lapping, mutually compatible signals without a clear origin. Indeed, Bentham’s first 
plan for the Panopticon had included an acoustic surveillance system with pipes 
leading from the cells to the central tower. The problem was that the prisoners 
could also hear what the inspectors were doing. Perhaps Bentham’s leaky network 
could offer a prototype for a more liberatory Internet. The strategy could be to 
refuse the metaphysical belief that information is light bestowed by a higher power, 
and to hack some leaks into the Panopticon. The following examples illustrate possi- 
ble strategies to level the playing field, suggesting updated metaphors that respond 


to the situations we encounter in the Internet age. 


Strategy 1. Go Underground 


The disorganized mob of bored geeks known as Anonymous have terrorized YouTube, 
the Australian government and the Mubarak regime, and have been named by CNN 
as a potential successor to WikiLeaks. Yet unlike political leaders who cherish the 
limelight, the primary weapon wielded by these guerrillas—apart from knowing 
enough JavaScript to be dangerous—is their studious application of technologies 
that render their identities invisible. They’ve poked leaks into security firms and 
governments, while remaining too dispersed across the ‘Dark Web’ to be pinned 
down themselves. 

The social architecture of Anonymous is not a watchtower that rises above its 
targets. Anonymous’ campaigns are not planned by a central committee but sug- 


gested by individuals and taken up or neglected by spontaneous volunteers. Some 
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Figure 2. Partial map of the Internet based on the 15 January, 


Jon Ippolito 


of its members have chosen as their 
emblem the cicada (Lipinski 2012), an 
insect that emerges from the soil every 
few years to make a ruckus and then 
buries itself underground again. A more 
fitting mascot might be the mycelium, 
the fungal root structure that can live 
thousands of years and extend nearly ten 
square kilometres.® (It’s no coincidence 
that maps of the Internet’s own ‘series of 
tubes’ plumbed by researchers like Bill 
Cheswick bear a striking resemblance 
to the mycelium’s rhizomatic branches, 
as figure 2 illustrates.’?) Despite their 
shadowy reputation neither myce- 
lia nor Anonymous are loners. Fungal 
roots stretch from tree to tree, passing 


nutrients and environmental informa- 


2005 data found on opte.org. Each node represents an IP address, 
with longer lines indicating longer delays in transmission between tion to an entire ecosystem of symbiotic 


the nodes. Courtesy of Wikimedia Commons, accessed 2 March, 
2014. http://en.wikipedia.org/wiki/File:Internet_map_1024 -_ 


flora. The hackers of Anonymous like- 


transparent.png. wise operate autonomously yet coalesce 


around popular causes promoted online 
in forums like 4chan. 


Strategy 2. Blind Them With Light 


In 1993 Jennifer Ringley of JenniCam had chosen the opposite strategy to hack the 
Panopticon: By setting up a Webcam and leaving it on 24/7 for seven years, she 
flooded the Web with unedited images of her everyday moments.’ While Ringley did 
not censor any of the ‘footage’ conveyed by her always-on Webcam, voyeurs keen 
on catching Ringley in a state of undress or in an intimate moment with a sexual 
partner were thwarted by the sheer volume of scenic data captured by the station- 
ary lens. In this case, complete visual access proved an effective if ironic barrier to 
prying eyes, forcing would-be Peeping Toms to confront quotidian moments in a real 
girl’s life, from brushing hair to sleeping to staring at a computer screen. 

One year before Ringley retired her Webcam, Hasan Elahi became Ringley’s 
successor, employing a more political strategy in response to US government 
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insinuations that he was a terrorist. 
Rather than conceal his private informa- 
tion from Big Brother, Elahi decided to 
leak everything about his life publicly, 
via a Website that obsessively docu- 
ments every plane he takes, every ham- 
burger he eats, every toilet he pees in. 
Unfortunately, while these floodlights on 
quotidian behaviour may have been radi- 
cal in their day, the subsequent launch of 
Facebook and Twitter showed that most 
netizens are happy to broadcast intimate 


details of their lives without political or 


Figure 3. Tracking data recorded by the author’s iPhone during a 
layover in the Philadelphia International Airport on 19 January, 2011. 
benefit of data-miners. The data was ‘unearthed’ using the free forensic tool provided by 


ack . researchers Alasdair Allan and Pete Warden (Fenton 2011). 
The British surveillance agency 


aesthetic motivation, to the commercial 


GCHQ’s secret harvest of Webcam 

images of millions of Yahoo users not sus- 

pected of wrongdoing between 2008 and 2010 provides a less deliberate, but equally 
delicious instance of Elahi’s visual noise drowning out the signal. Predictably 
enough, the project title, Optic Nerve, embodied yet another panoptic metaphor. The 
neural allusion in this codename aptly reflects the networked aspect of the project, 
which enabled spooks to sniff out and capture Webcam data packets as they routed 
through the Internet. The project’s ambition to emulate the ‘optical’ capabilities of 
a human spy might be forgiven since the word spy means ‘to look at’; yet the facial 
recognition software deployed by Optic Nerve failed to anticipate the large quanti- 
ties of Webcam flesh that wasn’t part of someone’s face.’ 


Strategy 3. Light Up Your Path 


Mobile technologies are like a Panopticon on wheels, which is why Richard Stallman 
won’t even carry a cell phone. German Green Party politician Malte Spitz had to 
take Deutsche Telekom to court to learn that his carrier had recorded his GPS posi- 
tion 35,000 times in a six-month period. Google Street View even captured a privacy 
advocate stepping out of his office at the Electronic Frontier Foundation to sneak 
a cigarette. Whereas the Panopticon surveilled its subjects from a central watch- 
tower, devices like tablets, smartphones, and fitness bracelets track their subjects 
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as they move through a dispersed global 
network of ski slopes, subways, and air- 
ports (see fig. 3). 

In an evocative response, Ze Frank 
invited people to use Google Street View 
to follow a remembered route they used 
to walk in childhood and then note down 
their emotional reactions. Frank’s inter- 
vention is more targeted than the ‘flood 
everything’ model of Ringley and Elahi, 
though of course his participants are 


Figure 4. John Bell, Octris (op 1 capriccio for bells and space), 2010, just giving Google more information on 
audiovisual virtual reality installation. 


themselves. Nevertheless Frank’s ‘if you 

can’t beat ‘em join ‘em’ strategy suggests 
that data mined by impersonal corporations can be repurposed to new ends—in this 
case, to remind citizens of the hometown roots from which a globalized economy has 
detached them. 


Strategy 4. Light Up a Decoy 


Heath Bunting’s anamorphic photos designed to fool security cameras and fake- 
identity services like Tracenoiser aim to chaff the network, distracting would-be 
data-miners with fake information. While Ringley and Elahi also injected additional 
information about themselves into the network, theirs was accurate; the ersatz 
homepages created by Tracenoiser are a pack of lies, each a different, algorithmi- 
cally created misrepresentation of its subject. But those who fly close to the sun 
have to take care not to get burned. The Security Camera Players were a group of 
actors who acted out scenes for the benefit of security guards in front of cameras in 
public spaces, until one of their lead actors was himself compromised by a hidden 
camera even he didn’t see. 


Strategy 5. Tunnel from Light to Dark 


John Bell’s Octris (see fig. 4) is a Virtual Reality version of Tetris that uses musical 
chords as cues for which falling puzzle pieces fit where.'° While originally designed 
with blind spatial engineer Nick Giudice as its audience, the work also trains sighted 
players to listen for acoustic matches by slowly dimming the light as they level- 
up the game. Instead of judging visually that an L-shaped puzzle piece should be 
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manoeuvred into an L-shaped hole, players who don Octris’ goggles and earphones 
must learn to move a piece by turning their head until the pitch associated with 
that piece finds other pieces whose notes complete a musical chord. To win the final 
level these viewers-turned-listeners must complete a line of the puzzle in complete 
darkness—a capability the game has prepared players for by slowly weaning them 
from visual cues onto auditory ones. It’s tough to complete that final level—just as 
it’s tough to relinquish our optical metaphors for information. Octris shows us that 


we can do both. 


Strategy 6. Light Up the Watchtower 


WikiLeaks may be the most infamous contemporary conduit of leaked information, 
but Steve Mann turned the tables on a store clerk in a much more personal way back 
in 1997. Wearing his own custom-made sunglasses with a built-in hidden video cam- 
era, Mann asked a store clerk if she was bothered by the nearby surveillance cam- 
eras. In response to her platitude ‘if you have nothing to hide, they shouldn’t bother 
you’, Mann took out his own camera—at which point she protested his invasion of 
her privacy. Thanks to his secret recording, these two contradictory moments are 
caught on the same video. 

Mann’s SafetyNet is perhaps the most progressive hack of the Panopticon to 
date. In his model, concealed Wearcams netcast continuous video from the wearer 
not to the entire world, but to a geographically dispersed band of compatriots who 
could ‘sousveille’ the wearer and intervene in the case of police brutality or personal 
injury. In giving one’s friends, rather than an official police force, the responsibility 
of looking after a person, SafetyNet is less a strategy of turning a floodlight on Big 
Brother than handing flashlights to one’s Little Brothers. 


Strategy 7. Dangle Something Shiny 


Like those deep-sea anglerfish, Natalie Jeremijenko’s project How Stuff Is Made 
offers baits that lure corporate mal-doers to step into the light of their own accord. 
This wiki produced by Jeremijenko’s students offers revelations about contemporary 
products that their manufacturers may not want public—such as the fact that shrimp 
farmers earn pennies per hour, American flags are made in China, and that the 
woman who stuffed my fortune into its cookie was quite possibly a Hispanic worker 
in Philadelphia. 

To set the trap, Jeremijenko’s students email the companies represented on the 


site and invite them to correct any factual errors. Marketing mavens can hardly 
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YOU CAN'T STOP TIME... 


But you can turn it back 

one hour at 2 a.m. on Oct. 28 
when daylight-saving time 
ends and standard time begins. 


Figure 5. A 2001 US public service advertisement that reminded 
people to adjust their clocks. Image courtesy Wikimedia 
Commons, accessed 2 March, 2014. http://en.wikipedia.org/wiki/ 
File:Daylightsavings.svg. 


resist the temptation of whitewashing 
a wiki to suit their PR needs, as studies 
of corporate influence in Wikipedia con- 
firm. In this case, the edits made to this 
wiki are visible in the page’s history, and 
Jeremijenko built the site with plans to 
highlight such changes in order to draw 


attention to corporate cover-ups. 


Strategy 8. Point Out Cracks 
in the Light Bulb 


The presumption that light conveys infor- 
mation about time is one of humanity’s 
longest-endured metaphysical equations, 
a carryover from an era in which all 
human activity was governed by circa- 
dian rhythms—that is, the sun’s traversal 
across a particular neighbourhood’s slice 
of the globe. As humanity’s reach spread, 
time zones were invented to patch the 
increasingly unreliable equation between 
light and time. In the nineteenth century, 
the telegraph and railroad expanded the 
need for a consistent nonlocal time; time 
zones solved the contradiction between 
the global time of networks and the local 
time of sunlight by insisting that ‘it’s 
10am your time but 7pm my time’. As 


technologies progressed, so did the cir- 


cumstances in which the imposition of time zones on a continuous sphere proved to 


be completely illogical. Circadian rhythms are different for different seasons, which 


is why Daylight Saving Time was introduced as a corrective in the early 1900s, 


screwing with humanity’s collective sleep cycle twice a year (and necessitating the 


kind of public awareness campaigns seen in figure 5). Circadian rhythms also vary 


by geography, which is why many nations chose not to obey Daylight Saving Time, 


and some closer to the poles such as Argentina, Iceland, and Russia have essentially 


The Panopticon is Leaking 215 


switched to it permanently (see fig. 6). 
As humanity expands its orbit beyond 
Earth, setting time by sunrise and sun- 
set ceases to be meaningful at all; satel- 
lites pass through many time zones in a 
day, and the clock effectively stood still 
for Apollo astronauts en route to and 


from the moon. 


For the ‘always on’ Internet, time 

Figure 6. Countries that adhere to Daylight Saving Time (blue: DST 
is used; orange: DST is no longer used; red: DST has never been 
years, annual and regional discrepancies used). Image courtesy Wikimedia Commons, accessed 2 March, 
2014. http://en.wikipedia.org/wiki/File:DaylightSaving-World- 
Subdivisions.png. 


zones are also an anachronism. In recent 


with time zones have played havoc with 
information technologies of all kinds. 
Apple’s iOS operating system failed to 
account accurately for Daylight Saving Time shifts in 2010, 2011, and 2013, causing 
alarms to wake European iPhone owners too late and Australian iPhone owners too 
early. In most cases these glitches were more inconvenient than life-threatening; 
however, in 2010 the US Food and Drug Administration warned that such arbitrary 
time shifts pose a grave danger due to the increasing dependence upon wearable 
medical devices for the timing of injections and similar treatments (FDA 2010). As 
an exercise in continuity, time zones also produce one of the most discontinuous 
artifices imaginable, the International Dateline—effectively a self-declared time 
machine in which stepping a metre east or west propels you forward or backwards 
24 hours in time." This awkward incongruity demonstrates the extent to which we 
are willing to give up bedrock assumptions about time in order to cling to cherished 
metaphors of light.'* 

Often among the first eager to explore metaphysical glitches, artists have 
exploited the anachronistic collision of time zones with global communication to cre- 
ative ends. Ken Friedman’s 1975 Fluxus performance In One Year and Out the Other 
instructed performers, ‘On New Year’s Eve, make a telephone call from one time 
zone to another so that you are conducting a conversation between people located 
in two years’.'3 In a similar paradigm updated to the Internet age, Curator Steve 
Dietz included in his 2006 exhibition Edge Conditions an installation by artists Jon 
Thomson and Alison Craighead consisting of a simple Webcam of the sky in Oceania 
shown in San Jose, artfully underscoring the contradictions inherent in applying the 
paradigm of light to the Internet under the title Light From Tomorrow." Artist Shawn 


216 


Jon Ippolito 


Brixey’s Epicycle likewise aims to expose the contradiction in linking Internet time 


to daylight, as Richard Rinehart explains: 


In this proposed new media work, Brixey would place cameras in each 
of the Earth’s twenty-four time zones, pointed at the horizon. In a cen- 
tral room, he would then display the live video feed on twenty-four 
monitors circling the viewer. The viewer would then be presented with 
a view of nature that is not possible in nature—a view in which the 
sun is always rising (on at least one of the screens around him in a 
perpetual sunrise/sunset). Epicycle uses technology in an attempt to 
marry two types of time: the ancient circadian rhythms of biology and 
geology and the newer global time where the sun never sets on the 
Net. (Rinehart and Ippolito 2014: 51-52) 


Optical metaphors can be tough to shake once ingrained in a language, as I’m 
reminded every time my blind neighbour concludes a visit with ‘T’ll see you later’. 
Yet we ignore their asymmetric metaphysics at our peril. An undue focus on the 
minarets where power shines brightly can blind us to the mycelia nourishing the 
gardens below—their potential to reinforce power or subvert it. Activists and artists 
of the Internet age offer networked approaches for turning back the Panoptic gaze. 
In an era of networks, we should be paying more attention to what we can’t see than 


what we can. 
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Notes 


1. Some of the concepts in this talk were originally introduced in Ippolito (1999: 26-27). 


2. Wikipedia has a good summary of PRISM’s history and scope. ‘PRISM (surveillance 
program)’, Wikipedia, http://en.wikipedia.org/wiki/PRISM %28surveillance _ 
program%29 (accessed 15 February, 2014). 


3. ‘empyre’ is an email mailing list frequented by new media artists and scholars, 
originally started by Melinda Rackham in Australia in 2002. http://www.subtle.net/ 
empyre (accessed 15 February, 2014). 


217 


218 Jon Ippolito 


4. 


10. 
11. 


12. 


13. 


14. 


From the standpoint of quantum physics, even light is capable of entanglement, an 
experimental result that calls into question whether light is a broadcast medium at all. 


Given our language’s bias toward the optical, it’s hard to avoid etymological oxymorons 
like ‘acoustic surveillance’—‘surveillance’ meaning literally ‘overseeing’. 


Paul Stamets (2005: 45, fig. 60) argues convincingly that a mycelium in Oregon is the 
largest-known organism on (or more precisely, under) the Earth. 


Bill Cheswick traces the routes of single information packets as they course along 
the frequently travelled backbones of Sprint and AT&T as well as the ‘back roads’ of 
regional phone systems. 


The Wikipedia page on Jennifer Ringley, in one of the site’s more memorable phrases, 
notes that ‘CNET hailed JenniCam as one of the greatest defunct websites in history’, 
citing the Internet Archive. 


The disappointed tone of GCHQ analysts reveals how out of touch they are with the 
social norms of contemporary netizens: ‘It would appear that a surprising number of 
people use webcam conversations to show intimate parts of their body to the other 
person’ (Ackerman and Ball 2014). 


For more information, see John Bell. 


This essay is based on a talk originally given over Skype from a location in Maine 
to the conference on the opposite side of the globe in Melbourne. The date in Maine 
happened to be the day before my birthday. Melbourne was fifteen hours ahead of 
Maine, and hence 7pm in Maine was 10am the next day in Melbourne. Thanks to 
the International Dateline, therefore, conference-goers in Melbourne were able to 
celebrate even though I was not, putting me in the awkward position of being 48 and 
49 years old at the same time. 


For very different reasons, Einstein made a similar choice to privilege light over time, 
and Special Relativity was the result. 


First performed on New Year’s Eve 1975-1976, calling from Springfield, Ohio forward 
to Nam June Paik, Christo and Jeanne-Claude, Dick Higgins, and George Maciunas in 
New York, then back to Tom Garver and Natasha Nicholson in California. Celebrated 
annually since then by telephone, telefax, and email. With Melbourne, Australia going 
into the New Year before most of the world, messages have gone back to the old year 
since 2008’ (Friedman 2013). It is yet another coincidence that Melbourne was the site 
of my teleconference mentioned in note number 11. 


I’m grateful to Craig Dietrich for reminding me about this piece. 
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He was commissioned for Channel 4’s Ghosts in the Machine, exhibited his work on 
Alive from Off-Center on PBS and Avance Sur Image on French Television and cre- 
ated a 5 part series on video art for Channel 4. He recently exhibited in Xi’an, China 
and at the Cathedral of St John the Divine in New York. As a cinematographer Terry 
shot for Apple during the making of Ridley Scott’s 1984 commercial; shot one of the 
first video to cinema releases in 1987; worked with analogue HD in the early ‘90’s; 
and shot four feature films. In 2007 he won an AHRC Creative Research Fellowship 
then in 2010 an AHRC Knowledge Exchange Fellowship at the University of Bristol. 
He is now Professor of Lens Based Arts at the University of the West of England. 


Jon Ippolito is an Associate Professor of New Media at the University of Maine, who 
specializes in doing the right thing in the wrong ways. Ippolito invents ways to build 
and sustain networks, a fact that often makes him unpopular with media monopo- 
lists, bureaucrats, and other apologists for hierarchic culture. Ippolito works with 
the Variable Media Network to devise new preservation paradigms to rescue digi- 
tal culture from obsolescence, with the Open Art Network to promote open archi- 
tectures for media art, and with the Interarchive working group to find net-native 


ways to connect online scholarship. He’s exhibited collaborative artworks at the 
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Walker Art Gallery, ZKM, and Harvard; curated shows for the Guggenheim on vir- 
tual reality and Nam June Paik; and written for the Washington Post, Artforum, and 
Leonardo. Ippolito’s collaborative architectures such as The Pool and ThoughtMesh 
have nabbed Wired headlines, while his book At the Edge of Art, co-authored with 
Joline Blais, offers an expansive definition for art of the 21st century. See www. 


three.org/ippolito 


Stephen Jones (born 1951, lives in Sydney, Australia) is an Australian video art- 
ist, curator, video engineer and conservator of long standing. He worked with Bush 
Video (1974-75), the Paddington Video Access Centre (1976-78), with Nam June Paik 
during his exhibition at the Art Gallery of New South Wales in Sydney (1976), as well 
as providing technical support for many major exhibitions including the Biennale 
of Sydney and Australian Perspecta from 1976 to 1985. From 1983 to 1992 he was 
the video-maker for the electronic music band Severed Heads. Between 1989 and 
1996 he worked as an engineer for several major video post-production and com- 
puter graphic production facilities. In 1996 he returned to making art and in 1998 
received a New Media Arts Fellowship from the Australia Council for the Arts. He 
provides technical support for artists, developing sensor-controlled systems for 
interactive video/DVD installations and physical immersion installations, as well as 
developing theoretical perspectives on interactivity, artificial intelligence and arti- 
ficial life. Since 2002 he has been researching the history of art and technology in 
Australia. This work has lead to his book, Synthetics: Aspects of Art and Technology 
in Australia, 1956-1975 (MIT Press, 2011). 


Carolyn L. Kane is the author of Chromatic Algorithms: Synthetic Color, Computer 
Art, and Aesthetics after Code (University of Chicago Press, 2014). Currently a post- 
doctoral fellow at Brown University (2014-2015), she received her Ph.D. from New 
York University in 2011. Her research fields include the history and philosophy of 


new media, electronic color, and digital aesthetics. 


Scott McQuire is an academic and writer with a strong interest in interdisciplinary 
research linking social theory, new media, art, and urbanism. He is the author of 
Crossing the Digital Threshold (1997), Visions of Modernity: Representation, Memory, 
Time and Space in the Age of the Camera (1998), Maximum Vision (1999), and most 
recently The Media City: Media, Architecture and Urban Space (2008) which won the 
Urban Communication Foundation’s 2009 Jane Jacobs Publication Award. He is also 
co-author with Peter Lyssiotis of the artist’s book The Look of Love (1998), co-editor 
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with Nikos Papastergiadis of Empires Ruins + Networks: The Transcultural Agenda 
in Art (2005) and with Meredith Martin and Sabine Niederer of the Urban Screens 
Reader (2009). Scott is Associate Professor and Reader in the School of Culture and 


Communication at the University of Melbourne. 


Daniel Palmer is Associate Dean of Graduate Research and Senior Lecturer in the 
Art History & Theory Program at MADA (Monash Art, Design & Architecture). He 
has a long-standing involvement with the Centre for Contemporary Photography in 
Melbourne, as a former curator and current board member. His publications include 
the books Twelve Australian Photo Artists (2009), co-authored with Blair French, 
and the edited volume Photogenic: Essays/Photography/CCP 2000-2004 (2005). His 
scholarly writings on photography have appeared in journals such as Photographies, 
Philosophy of Photography, Angelaki and Reading Room, and he also contributes to 


art magazines including Art & Australia and Frieze. 


Christiane Paul is Associate Professor at the School of Media Studies, The New 
School, and Adjunct Curator of New Media Arts at the Whitney Museum of American 
Art. She has written extensively on new media arts and lectured internationally 
on art and technology. Her recent books are: Context Providers - Conditions of 
Meaning in Media Arts (Intellect, 2011; Chinese edition, Beijing Beepub Media & 
Culture Publishing Co., 2012), co-edited with Margot Lovejoy and Victoria Vesna; 
New Media in the White Cube and Beyond (UC Press, 2008); and Digital Art (Thames 
and Hudson 2003; expanded new edition 2008). As Adjunct Curator of New Media 
Arts at the Whitney Museum of American Art, she curated several exhibitions— 
including Cory Arcangel: Pro Tools, Profiling (2007), Data Dynamics (2001) and the 
net art selection for the 2002 Whitney Biennial—as well as artport, the Whitney 
Museum’s website devoted to Internet art. Other recent curatorial work includes 
The Public Private (Kellen Gallery, The New School, Feb 7-Apr 17, 2013); Eduardo 
Kac: Biotopes, Lagoglyphs and Transgenic Works (Rio de Janeiro, Brazil, 2010); 
Biennale Quadrilaterale (Rijeka, Croatia, 2009-10); Feedforward - The Angel of 
History (co-curated with Steve Dietz; Laboral Center for Art and Industrial Creation, 
Gijon, Spain, Oct 2009); INDAF Digital Art Festival (Incheon, Korea, Aug 2009); and 
Scalable Relations (Beall Center for Art and Technology, Irvine, CA; as well as galler- 
ies at UCSD, UCLA and UCSB, 2008-09). Dr. Paul has previously taught in the MFA 
computer arts department at the School of Visual Arts in New York (1999-2008); the 
Digital+Media Department of the Rhode Island School of Design (2005-08); the San 
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Francisco Art Institute and the Center of New Media at the University of California 
at Berkeley (2008). 


Alvy Ray Smith cofounded two successful startups: Pixar (sold to Disney) and 
Altamira (sold to Microsoft). He was present at the beginning of computer graphics 
at Lucasfilm, the New York Institute of Technology, and the Xerox Palo Alto Research 
Center. He was the first Graphics Fellow at Microsoft. He has received two techni- 
cal Academy Awards, for the alpha channel concept and for digital paint systems. 
He invented, directed, originated, or was otherwise instrumental in the following 
developments: first full-color paint program, HSV (aka HSB) color model, alpha 
channel, the Genesis Demo in Star Trek II: The Wrath of Khan, The Adventures of 
André & Wally B., first Academy-Award winning computer-generated short Tin Toy, 
first computer-generated film Toy Story, Academy-Award winning Disney animation 
production system CAPS, and the Visible Human Project of the National Library of 
Medicine. He was a star witness in a trial that successfully invalidated five patents 
that threatened Adobe Photoshop. Dr. Smith has a PhD from Stanford University and 
an honorary doctorate from New Mexico State University. He is a member of the 
National Academy of Engineering and a Fellow of the American Association for the 
Advancement of Science. He has published often in theoretical computer science and 
computer graphics, and holds four patents. He retired in 2000 to devote time to the 
emerging art form of digital photography and to scholarly genealogy, to which he 
has contributed two award-winning books and half a dozen learned journal papers. 
He is Trustee Emeritus of the New England Historic Genealogical Society, Boston, 
and a Fellow of the American Society of Genealogists. He has won many awards, 
speaks widely, and is now writing a book, A Biography of the Pixel, a laymen’s guide 


to modern media. For more see alvyray.com 


Darren Tofts is Professor of Media and Communications, Swinburne University 
of Technology. He is the author (with artist Murray McKeich) of Memory Trade. 
A Prehistory of Cyberculture (Sydney: Interface Books, 1998), Parallax. Essays on 
Art, Culture and Technology (Sydney: Interface Books 1999) and Interzone: Media 
Arts in Australia (Thames and Hudson: Sydney, 2005). With Annemarie Jonson 
and Alessio Cavallaro he edited Prefiguring Cyberculture: An Intellectual History 
(Power Publications/MIT Press, 2003) and with Lisa Gye edited Illogic of Sense: 
The Gregory L. Ulmer Remix (Colorado: Alt-X Press, 2009). His most recent book is 
Alephbet: Essays on ghost writing, nutshells and infinite space (Prague: Litteraria 
Pragensia, 2013). 
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Cathryn Vasseleu teaches animation at the University of Technology, Sydney. She 
is author of Textures of Light: Vision and Touch in Irigaray, Levinas and Merleau- 
Ponty (Routledge, 1998), editor of Jan Svankmajer’s Touching and Imagining: An 
Introduction to Tactile Art (I. B. Tauris, 2014), and writer/director of animated exper- 


imental films including De Anima (1991). 
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